There are many options and alternatives. But I can’t answer which of any of these would be appropriate for a particular environment. Not without knowing a bit more about the requirements.
File shares? What would you have as an alternative to SMB? Ponder what users are presently working with for file shares, and where most users are headed with their products and configurations. AFP isn’t it, not is NFS, nor the other alternatives.
As for SMB, Microsoft publishes the specs. There been SMB updates and changes and new versions. Microsoft is a vastly different company than it was even a decade ago, and the computing market itself is very different from the era that you’re citing, too. But I digress.
Alternative operating systems? The BSDs (prolly FreeBSD, maybe OpenBSD), Arch certainly, Centos/Fedora (these choices particularly possibly for some eventual migration to a RHEL configuration with support), etc. Genode is very new but also quite interesting, DragonFly is more established but interesting, etc. Arch rolls forward pretty quickly, but also seeks to be easy to manage. The BSDs and particularly OpenBSD target security. If you’re seeking a production solution or a more turnkey server product option, Synology has a number of plug-ins for their plethora of NAS boxes and those plug-ins can provide directory, mail, DNS, and many other network services. And yes, Synology can host SMB and NFS file shares, among other services. And tends to be easy to manage. What works best here is extremely dependent on your experience, your expectations and plans, and of course on your budget.
As local back-end servers? Many folks use Microsoft Windows Server and Active Directory for directory, DNS and other network services. Others are using hosted services as the back-end, whether that’s thr hosted AD and hosted Exchange Server offerings or otherwise, or a move to profiles and provisioning (of which macOS with Server.app will continue to support, per that Apple document) and with some assistance and use of Apple iCloud.
As for where Apple is headed with macOS and iOS, we’re not in a position to speculate about that here in the Apple forums. We’ll learn more with the upcoming Spring (northern hemisphere) server-related announcement, and with WWDC.