Here's what I did overall to get it working and it is now working on all my Big Sur implementations (4):
- In DSM>Control Panel>File Services>SMB/AFP/NFS select SMB Advanced Settings and set maximum SMB Protocol to SMB3 and minimum to SMB1. I kept AFP enabled even though it has been deprecated in Big Sur. In DSM>Control Panel>File Services>Advanced I ensured that Bonjour discovery was enabled for both SMB & AFP.
- In DSM I deleted the existing shared folder that I designated for Time Machine and created a new one. I went back to DSM>Control Panel>File Services>Advanced Settings>Set Time Machine Folders and picked the new one I just created.
- In DSM>Shared Folder>Permissions I ensured that only one (1) of the users I created had privileges to the newly created Time Machine folder.
- In DSM>Control Panel>Users I went through each user to ensure none other than the one (1) I had selected for Time Machine backups had permissions to that shared folder, and I ensured that user had an unlimited quota.
- On my Big Sur Macs I totally stopped the Spotlight Indexing Service altogether (in Terminal: sudo mdutil -i off).
- On my Macs I also uninstalled the Sophos AV as well.
- On my Macs I removed the existing Time Machine destination and unchecked automatic backup.
- I then shutdown each Mac and did an SMC reset (not NVRAM).
- After rebooting a Mac I then selected the new destination on on my Synology device using the credentials of the sole user with those permissions.
But before doing any of that take a look at the Connections log on your DSM (DSM>Main Menu>Log Center, and select Connections from the drop down menu. Check if when a Time Machine backup initiates whether you have more than one user attempting the connection. For example, I noted that when my backups failed I would have two (2) users that had privileges to that shared folder attempt to connect nearly simultaneously from the same Mac even though I had only supplied one user credentials when selecting the share in Time Machine setup. I have no explanation or even speculation as to why that was happening.
I suspect there is a bug lurking in Big Sur regarding Apple's choice of APFS Time Machine backups that causes it to not play well with NAS devices, or in particular Synology NAS devices. It won't surprise me in the least if this is addressed in an 11.1 release. Everyone knows the 11.0.1 release was rushed to coincide with the M1 Macs show, and may not still be ready for primetime. I have a Mac I use as an Apple Music/TV/Plex server that relies on a 24tb SoftRAID RAID5 array that I cannot transition to Big Sur until SoftRAID is able to release an update (they claim their updated driver is built into the Big Sur release, but users are reporting issues and the client is beta and will be for some time). I'm just not going to risk that machine to some fatal bug that totally borks a 24tb array.