The firmware version is 5.16.105. So the protocol could be (and should be, right?) SMB, but why the page in the NAS UI reads that AFP is automatically selected?
Yes, it should be running a version of Samba that supports that specification.
No idea except they are ignorant of the change to SMB (and that their NAS isn't actually supported over AFP--never has been).
Also, Barney, you say Time Machine over AFP is no good (you have answered that before on another NAS question), can you tell how that will show? Does it show right away, in some way? (A "this ain't gonna work message.") Or on a day in the future when we need a certain backup (with a nasty message: "we never backed that up for you")? Or by installing a broken backup?
In the past (pre-Big Sur), the problems usually only manifested when you needed to Restore and it just wouldn't work.
Other manifestations included frequent messages to the affect, "the backup cannot be verified and a new full backup is required." That happens because the specification adds in error checking requirements to ensure the backup is correctly written. Netatalk (the hack of AFP that all NAS use) most likely does not contain those error checking and validation cues that Apple implemented in AFP to make Time Machine work over a network.
I've seen several posts now on Big Sur where they can't even get a full backup successfully, or the verification message happens quite often. I think there was one where SMB didn't work (spec not supported on that NAS) so they switched back to AFP and it worked for them. No idea if the same old problem of not being able to restore will appear.
The WD website (your link Loner) says Time Machine is supported on Big Sur and below. And then they say nothing about AFP and SMB as if it's not an issue at all.
They are either ignorant of how it is supposed to work or are purposely obfuscating the issue to not make it obvious their other products won't actually work correctly. It is somewhat esoteric, so they would not need to specify the precise support except that they've been selling hardware claiming support when it never really correctly supported Time Machine and was never approved as a backup destination by Apple.
Various websites surmised that Apple never supported a NAS for Time Machine backups because they were selling their own NAS hardware (Time Capsule), but that wasn't really the reason. Time Machine over a network needs extra error checking that was only available in Apple's AFP. Since Apple never licensed AFP, they had no confidence that the NAS AFP implementation supported those features. When Apple switched over to SMB, they published a specification for SMB to make network Time Machine reliable over SMB (much as they implemented in their own AFP).
As per the Backup Disks document LonerT linked, the NAS manufacturers should be indicating support for the "Time Machine over SMB specification," but that would hint at the lack of support they have for AFP. As I implied earlier, this shouldn't be necessary, but since they played fast and loose with the "we support Time Machine" schtick, it needs to be stated so that you are actually buying a product that meets Apple's guidance.
Catalina is on APFS, does that help to determine if the right protocol is used (as in: 'it wouldn't use AFP if the Catalina disk is APFS formatted')?
macOS will not share out an APFS volume over AFP. The NAS file server can share anything it wants over any protocol it supports. There may be technical limitations that prevent sharing APFS over AFP, but I imagine it is just Apple not wasting resources making it possible in a dead protocol. It could even be that it would work just fine, but the cost of testing to verify that would be prohibitive.
The communication protocol should not dictate how the data is written to or read from the file server, so the file system format should not matter in the least.