If ssh access is enabled on the target system (client or server) and the SACL is not blocking access (server), then password-based access is available via ssh. sftp operates atop ssh.
Server settings are in Server.app in the Settings section, client systems enable remote inbound ssh access in the System Preferences > Sharing section.
Generating ssh keys uses the key-gen tool. On macOS, the command-line command is ssh-keygen. I'd skip that for now, at least until the password-based logins are working. Once that's all working, somebody here can point you to discussions of generating keys and of transferring and adding the public keys into the target system.
If the above isn't it? What's going on here? I don't know. The more this thread extends, the more I would strongly recommend against hosting Windows backups locally on this server, as I'm concerned that there might be or are additional security problems exposed or introduced here; the trade-offs involved here are not good ones, around exposing this server in its current state, and given the familiarity with macOS in these environments. A whole lot can go wrong with network-exposed systems and security, and the results are often Not Fun.
