SSH command line using Terminal not working to connect to iPhone Hotspot on macOS Sonoma

Question

Level 1

5 points

SSH command line using Terminal not working to connect to iPhone Hotspot on macOS Sonoma

When using an iPhone as a hot spot with a Mac, the ssh command line (and using ssh from Terminal) will not connect to remote hosts.

This is a MacOS issue, not a hot spot issue. Here’s why:

When I use the third party app, SecureCRT, on my Mac, it operates just fine for SSH sessions to the exact same remote servers that the ssh command line (and Terminal with ssh) refuse to connect to.

I am able to watch packet flow on the remote server and verify that the MacOS command line ssh does not even try to connect. Packets don’t even leave the Mac.

Given this same behavior is NOT seen when using other mobile/cellular (purpose built hotspot device, or a Peplink mobile router) via Wifi, this appears to have been a deliberate choice, or a specific bug, when it comes to the hot spot connectivity function of Macs and iOS/iPadOS devices.

In prior posts, I’ve seen Apple and other users try to help diagnose this as a hot spot settings issue on the iPhone. My use of a third party app disproves this. I am curious, if this was done on purpose, to understand why an effort was made to intentionally break ssh, and possibly other Mac functionality, when using an iPhone/iPad as a hot spot. If it’s not intentional, I’ll look forward to a bug fix showing up in MacOS someday.

[Re-Titled by Moderator]

MacBook Pro 13″, macOS 14.5

Posted on May 21, 2024 12:40 PM

Reply

Answer 1

Top-ranking reply

dsnh03 Author

Level 1

5 points

May 22, 2024 11:16 AM in response to MrHoffman

Thanks for providing a useful way to diagnose. Obviously, I've changed the host names, and target ports, for security reasons in the outputs below. Note that the results are not affected by using port 22, 2000, o any other port choice. Keep reading to the end of this post, as the issue isn't related to TCP connectivity at all, as I've demonstrated, but rather an issue with the ssh client.

When going over Wifi or Ethernet, I get:

% ssh -vvv -p 823 host.example.net

OpenSSH_9.6p1, LibreSSL 3.3.6

debug1: Reading configuration data /Users/XXXXX/.ssh/config

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files

debug1: /etc/ssh/ssh_config line 54: Applying options for *

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/XXXXX/.ssh/known_hosts'

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/XXXXX/.ssh/known_hosts2'

debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling

debug3: channel_clear_timeouts: clearing

debug1: Connecting to host.example.net port 823.

debug1: Connection established.

And when over iPhone hot spot I get:

% ssh -vvv -p 823 host.example.net

OpenSSH_9.6p1, LibreSSL 3.3.6

debug1: Reading configuration data /Users/XXXXX/.ssh/config

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files

debug1: /etc/ssh/ssh_config line 54: Applying options for *

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/XXXXX/.ssh/known_hosts'

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/XXXXX/.ssh/known_hosts2'

debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling

debug3: channel_clear_timeouts: clearing

debug1: Connecting to host.example.net port 823.

ssh: connect to host host.example.net port 823: Undefined error: 0

Now, using netcat, over hot spot, it's clear TCP connects:

% nc host.example.net 823

SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.11

Because nc shows the prompt back from the target server immediately, we know the TCP session is established and SSH daemon on the target server is waiting for commands.

I can see, using tcpdump on the target server, that the nc example above results in packets and the ssh does not, both while using the iPhone as hot spot.

I do not see a routing issue as the cause here. I've looked at the routing tables on my mac.

What I do see, that offers a clue, relates to DNS:

While nc works fine with the host name, looks it up and all works properly, ssh DOES work with IP address, but fails with hostname. This is not an issue on Ethernet or non-hot-spot Wifi. As such, it appears the difference between the hot spot case and the wifi/Ethernet, is DNS resolution ONLY on the ssh command line. This is certainly curious. Before jumping to conclusions about DNS problems on the Mac, please re-read this paragraph, and note that other command line functions resolve the DNS correctly, as do Safari, Mail.app and others. This relates somehow only to something ssh is doing, and that error it produces:

ssh: connect to host host.example.net port 823: Undefined error: 0

Reply

Answer 2

dsnh03 Author

Level 1

5 points

May 23, 2024 7:46 AM in response to etresoft

Apple uses the OpenSSH client. How they modified it for use in their heavily-modified Unix variant is really the root issue, as they have a DNS lookup bug in how that one utility is looking up hostnames. Using SSH with an IP address to the same target works. Using hostname with nc, ping, and every other utility I have tried works fine. Sometimes the issue really is a bug.

Reply

Answer 3

etresoft

Level 9

57,518 points

May 22, 2024 12:21 PM in response to dsnh03

dsnh03 wrote:

Before jumping to conclusions about DNS problems on the Mac

No need to worry. 😄

This relates somehow only to something ssh is doing, and that error it produces:

ssh: connect to host host.example.net port 823: Undefined error: 0

You already said you couldn't connect.

Have you considered IPv6 problems? Apple's a stickler for IPv6 support. So any modern tools that properly use Apple's networking APIs get proper IPv6 support out of the box. But if your hotspot doesn't support IPv6, then you've got a problem, but only with IPv6-aware apps. Older apps, or things ported from other operating systems, probably don't know anything about IPv6 so nothing has changed as far as they're concerned.

And DNS could be involved. It could be caching the IPv6 address and then refusing to connect when IPv6 goes away or becomes a dead end. Or perhaps the hotspot simply drops any IPv6 packets at the door.

Reply

Answer 4

dsnh03 Author

Level 1

5 points

May 23, 2024 7:42 AM in response to etresoft

The issue is isolated to the ssh command line utility and its handling of host name lookup. Please re-read the messages in the thread to see where I've isolated the issue there. The issue is NOT a carrier issue. It's been tested with multiple carriers over regular WiFi and over Apple's hot-spot service between mac and iPhone and iPad. Specifically, the iPhone and iPad are on different cellular carriers, and the non-Apple hot spots are on the same carriers as the iPhone and iPad, so all combinations were indeed tested. It's simply a bug in Apple's port of the OpenSSH client.

Reply

Answer 5

MrHoffman

Level 10

150,537 points

May 21, 2024 3:09 PM in response to dsnh03

dsnh03 wrote:

When using an iPhone as a hot spot with a Mac, the ssh command line (and using ssh from Terminal) will not connect to remote hosts.

This is a MacOS issue, not a hot spot issue. Here’s why:

When I use the third party app, SecureCRT, on my Mac, it operates just fine for SSH sessions to the exact same remote servers that the ssh command line (and Terminal with ssh) refuse to connect to.

I am able to watch packet flow on the remote server and verify that the MacOS command line ssh does not even try to connect. Packets don’t even leave the Mac.

i’m unclear how you are observing the source Mac from the destination server. If your equipment supports it, you can mirror the Mac switch port and watch the traffic, as the remote server cannot observe issues at the source Mac, or the Hotspot, or within the mobile network.

You’ll also want to use traceroute among other debugging tools, and command-line ssh and also ssh --vvv as part of your connectivity testing.

Reply

Answer 6

etresoft

Level 9

57,518 points

May 21, 2024 3:59 PM in response to dsnh03

dsnh03 wrote:

You might look up SecureCRT. Not using an ancient anything. Using SSH with modern cyphers.

I did. I saw that it was cross-platform. That almost certainly means it is using legacy networking APIs and probably its own secure certificate database. That's common for Linux and open source tools ported to the Mac.

If third party OS stuff were involved, it would also affect regular Wifi connections.

Not necessarily, for reasons that I explained above.

But what do I know? Only 40+ years of Unix/Linux internals, IETF standards work on protocols, NAT and more, plus a rather extensive software QA background.

But what's missing from that list? macOS, AKA iOS. It's totally different now. The "macOS is BSD" story is just a myth. Unix and Linux didn't have the biggest corporations in the world hacking on them 24/7 like Apple does. Modern networking on iOS/macOS is significantly different that what you might be familiar with.

Reply

Answer 7

MrHoffman

Level 10

150,537 points

May 21, 2024 4:30 PM in response to dsnh03

What does a port scan of your server show from the client via Hotspot, as compared with what the server should show for open ports?

What does the traceroute from client to server show? Headed on the right path?

What does command-line ssh --vvv show for its connection activity?

Whether this is a local routing issue, or that ssh client getting tangled by network routing, or some local network device usurping the Hotspot route?

Reply

Answer 8

etresoft

Level 9

57,518 points

May 21, 2024 2:47 PM in response to dsnh03

Just because one third party app works doesn't mean anything. Most likely, that app uses some ancient/foreign network stack that is incompatible with modern networking. Some other problem is breaking the networking, but through random luck, doesn't affect this specific app. The most likely cause, as always, is 3rd party system modification software. Such apps are extremely popular and cause all kinds of problems. They are what keeps this forum running.

Reply

Answer 9

dsnh03 Author

Level 1

5 points

May 22, 2024 12:54 PM in response to etresoft

I have already considered IPv6 issues. The target host has no AAAA record. It does not resolve an IPv6 address.

Note once again, nc has no issue resolving the host name to an A record and connecting. This points to an issue with how ssh is handling the lookup, and handling it differently when using an iPhone or iPad as hot spot vs. any other wired or wireless network.

The "hot spot" in this case is an iPhone 15 Pro, running the latest iOS. The Mac is running Sonoma, latest update.

The person typing this has much clue about how TCP/IP works, how DNS works, how resolvers handle IPv4 and IPv6.

Reply

Answer 10

MrHoffman

Level 10

150,537 points

Jul 10, 2024 8:07 PM in response to Ek Jatt

Ek Jatt wrote:

1) Can anyone confirm that they are able to connect to Mac devices over internet with current macOS.

2) then test it using hotspot (iPhone) on both end, latest iOS and post here if they are able to connect.

1: is commonplace, though does typically require either a site-to-site VPN, or port forwarding.

Gather some evidence for the claims about ssh and carrier hotspot here (ssh -vvv logs, firewall logs, server logs, etc), and then contact Apple Support and present your case.

Reply

Answer 11

MrHoffman

Level 10

150,537 points

May 22, 2024 4:15 PM in response to dsnh03

dsnh03 wrote:.

The "hot spot" in this case is an iPhone 15 Pro, running the latest iOS. The Mac is running Sonoma, latest update.

Tried a different Hotspot from a different carrier?

Reply

Answer 12

dsnh03 Author

Level 1

5 points

May 21, 2024 3:07 PM in response to etresoft

You might look up SecureCRT. Not using an ancient anything. Using SSH with modern cyphers. Thanks for your response.

If third party OS stuff were involved, it would also affect regular Wifi connections. But what do I know? Only 40+ years of Unix/Linux internals, IETF standards work on protocols, NAT and more, plus a rather extensive software QA background.

Reply

Answer 13

dsnh03 Author

Level 1

5 points

May 21, 2024 4:05 PM in response to etresoft

I have SecureCRT using the same keys I use with command line ssh. As for protocol stacks, they likely do use their own. I'd still love you to explain, however, how the ssh command line explicitly fails to work over Wifi to an iPhone, while it works fine with Wifi to a Peplink router or other Wifi connection to a non-Apple device.

I'll also note that when I use netcat to test connection to the ports I use for SSH (both standard and otherwise), netcat connects. When I use ssh, no packets leave the mac at all. This isn't an issue of protocol differences. It's a matter of not even trying.

Reply

Answer 14

etresoft

Level 9

57,518 points

May 21, 2024 4:29 PM in response to dsnh03

dsnh03 wrote:

I have SecureCRT using the same keys I use with command line ssh. As for protocol stacks, they likely do use their own. I'd still love you to explain, however, how the ssh command line explicitly fails to work over Wifi to an iPhone, while it works fine with Wifi to a Peplink router or other Wifi connection to a non-Apple device.

It's not about the keys. It's about the multiple network stacks. I can't explain how your network is configured. I can tell you that you're looking in the wrong place, but that's all.

It's a matter of not even trying.

Why not? That's the question.

Here's the problem in a nutshell, in your own words:

This is a MacOS issue

...

I am curious, if this was done on purpose, to understand why an effort was made to intentionally break ssh, and possibly other Mac functionality, when using an iPhone/iPad as a hot spot. If it’s not intentional, I’ll look forward to a bug fix showing up in MacOS someday.

First off, there is no "MacOS" or "macOS" or however you want to spell it. There is only iOS. The Mac platform has some funky hacks designed to make most legacy apps continue to work. Don't rush to judgement when you stumble upon one of those hacks.

It may be possible to determine where the problem lies, but not when you've already convinced yourself that the answer is pre-determined. There will be no bug fix showing up. Over time, Apple removes more and more of those funky hacks. Eventually, your SecureCRT app will stop working.

Regardless, it is always a bad idea to "look forward to a big fix". Fix fixes like that never, ever arrive. You'll need to file a detailed bug report reproducing the issue, get a few thousand of your best friends to do the same, and maybe you might get a meaningful response regarding why it won't be fixed.

Reply

Answer 15

etresoft

Level 9

57,518 points

May 22, 2024 2:39 PM in response to dsnh03

dsnh03 wrote:

The "hot spot" in this case is an iPhone 15 Pro, running the latest iOS. The Mac is running Sonoma, latest update.

Maybe the problem is the carrier. I've never had any problems with ssh over a hotspot.

Reply