there was a problem connecting to server error

I'm assuming .178 is the iMac having issues ... correct?

If so, looks like the handshake is occurring at packets 69-71, but a FIN is being immediately sent from the iMac to start the termination of the communication session with the server in 72. Basically, the iMac is saying I don't have anything else to send you. I would have expected that the start of the SMTP process would be have been the next packet instead.

Packet 77 appears to be a Dup ACK. Possibly there are more of them later in the capture. If so, they are typically sent during a TCP fast retransmission sequence because (iMac in this case) did not receive an expected ACK from the server.

... and what is one cause of lost packets/retransmissions? A faulty connection between two devices on the network. If you ping the server from the iMac are there any packets being dropped and/or any excessive delays seen? RTT times should be around 1-1.5 ms on a local network. The other potential issue would be mis-configured Ethernet interface ... assuming that the iMac or the server (or both) connect to the local network by Ethernet.

Network is based on a default 192.169.1.1 gateway.

Did you mean to write 192.168.1.1 here? As that would be a local address.

OS X/macOS has supported all variants (1-3) of SMB since Yosemite. However, Microsoft dropped support for SMB 1.x with Windows 10. This may be simply a SMB configuration issue on the Win 10 server. I'm not sure if this is related to it only being the "Home" edition vs. the "Pro" one, but it could be possible. I do know that the Home edition is somewhat limited networking-wise.

A data capture between these device would go a long way to reveal which SMB protocol is negotiated ... or, at least, attempted. If you're familiar with Wireshark (or equivalent) you can do this yourself. There is also a method to enable SMB 1.x on Windows 10, should it be required. However, I'm not necessarily recommending it.

Ref:

• Troubleshooting Microsoft SMB connect Issues with Wireshark -NetworkDataPedia
• SMB: Troubleshooting Guide - Microsoft TechNet
• Turn off packet signing for SMB 2 and SMB 3 connections - Apple Support

Thanks for hanging in there.

Do both the iMac and the server connect directly to the Linksys router ... or are there any Ethernet switches also involved? If so, what is the make(s) & model(s) of the/these switch(es)?

Does your ISP actually provide you with IPv6 service? If not, then it would be better to reconfigure your network clients for "IPv6 Link-local" only mode. This will still allow for IPv6 traffic, but only locally. A number of macOS services rely on IPv6 for network discovery.

Again, if you can provide me with your iMac's Ethernet settings that would be helpful.

I would also like if you can run a few ping tests for me, and provide the results. Please use the following ping commands, in a Terminal window, on your Mac:

1. ping -c3 127.0.0.1
2. ping -c3 <your Linksys router's LAN-side IP address>
3. ping -c3 <your server's IP address>

The "best" possible results will result in no packets being dropped, but I'm also interested in the average RTT values for each of those pings.

Some potentially helpful Wireshark details for SMB troubleshooting and what communications between the Mac and the Win 10 server to look for:

1. The Mac will first perform a "TCP handshake" to establish communication with the server.
2. The Mac will send a "Negotiate Protocol Request" to the server to let it know what SMB dialects the Mac supports.
3. The server will respond with a "Negotiate Protocol Response" indicating what SMB dialects it supports, as well as, other details like if signing is enabled.
4. The Mac will send another "Negotiate Protocol Request" to the server to let it know which dialect and which options is want to use.
5. The server will respond with another "Negotiate Protocol Response" to let the client know which dialect it will actually use.
6. The Mac will then agree to use that dialect, and then, asks the server "What authentication token that you want to use?" This starts the authentication phase. This is where the user account is validated to allow access to the server.

Give the following setting a try:

• Configure: Manually
• Speed: 1000baseT
• Duplex: full-duplex
• MTU: Jumbo (9000)
• AVB/EVB Mode: disable

It's getting late here so I will be signing off, but I do have a few more questions for you if you don't mind.

Since they are close, are both connected to the same Ethernet switch? Actually, a bit more information on the network path between the iMac and the server would be helpful. That is, what networking hardware lies between the two to make the full connection? For example, do they both directly connect to an Ethernet switch like I mentioned or do they each connect to an Ethernet wall port, and in turn, an Ethernet run to a central Ethernet patch panel, to a switch, and then, to a router?

I would also like to know the Ethernet settings on your iMac from In System Preferences > Network > Ethernet > Hardware tab. Does the iMac (& server) both assigned static IP local addresses. Are they configured for IPv6? Is the iMac configured as a DHCP client? Is the iMac configured to only use Ethernet and is Ethernet at the top of the list on the System Preferences > Network window?

Sorry, for all the questions, but they may reveal some clues to what may be the culprit.

So how is your iMac connected to this network? “Hardwired” or WiFi?