It seems you have set up properly.
I haven't used remote SMB connections lately, so I decided to try it again. Connecting from a Yosemite machine to Catalina worked and from Files app on an iPad to Catalina worked as well.
Curiously, If I try to establish a connection with myself (unlike scenario) using the hostname it doesn't work. Using the IP address it does work. I suppose it's because the system refuses to connect with itself. In fact, VNC gives me an error (You cannot control your own screen) when using the hostname and works flawlessly when using the plain IP address.
In my case my local network has a double NAT (Modem and Router). It seems to me that it's your case, too – if true this should be the "scheme" (addresses are not real):
Internet – External IP 123.456.789.000
|
Modem – Internal IP 192.168.1.1 – Port forwarding (22, 137, 138, 139 and 445) to WiFi Router 172.16.0.1
|
WiFi Router – Internal IP 172.16.0.1 – Port forwarding (22, 137, 138, 139 and 445) to iMac 172.16.0.50
|
iMac – Internal IP 172.16.0.50
At least the IP connection should work: smb://123.456.789.000
Speaking of FTP: for SFTP connections (SSH File Transfer Protocol) you have to use a proper FTP client which supports SFTP. Only plain FTP is supported by Finder.