John Galt's reply is neither absolutely accurate nor absolutely inaccurate.
The original Time Machine feature supported using either a local HFS+ formatted hard drive or a genuine Apple Mac running Server.app and acting as a Time Machine Server, or an Apple Time Capsule also acting as a Time Machine Server. The later two options i.e. Time Machine Servers required using an AFP network share.
Most NAS makers use an open source program called Netatalk to provide their AFP service and the Netatalk team have also added the ability for it to act as a Time Machine Server. However where John Galt is correct is that Netatalk have done this via much guesswork as to what Apple are doing and Apple also have a nasty habit of moving the goal posts each time they update the Mac operating system. Therefore Netatalk cannot be guaranteed to be 100% compatible. Worse still the NAS makers may not have the latest version of Netatalk in their products. Despite this Time Machine over AFP to a NAS works well enough for many people.
On the topic of backups you should always test they can be successfully used to do a restoration and should consider having two different backup systems e.g. a clone as well as a Time Machine backup.
Where John Galt is however being overly pessimistic is that Apple do not forbid NAS makers from providing Time Machine capability and for most people it has historically been 'adequate'. Furthermore in more recent times Apple have actually been more forthcoming and supportive of the efforts of NAS makers to do this. Read-on…
As you have been trying it is now possible in theory to do Time Machine backups to a Time Machine Server via SMB instead of AFP. This is officially supported using Apple's Server.app but not officially supported on a Time Capsule.
NAS makers use another different open source program called SAMBA to implement SMB support and the SAMBA team have been working on incorporating the official requirements published by Apple on how to do this. See https://developer.apple.com/library/content/releasenotes/NetworkingInternetWeb/T ime_Machine_SMB_Spec/
However as far as I can tell the SAMBA team have not yet finished this work and therefore even if a NAS maker has included the very latest version of SAMBA it will not yet have the required changes. As a result SAMBA is not yet going to have solved any issues still remaining. See https://bugzilla.samba.org/show_bug.cgi?id=12380
So the current situation is that Time Machine over AFP to a NAS is in general possible and in many peoples experience 'works'. Time Machine over SMB to a NAS however is an unknown entity and very likely to still have issues and I would therefore even myself not yet rely on it.