Can I roll back the operating system to old version with my time machine backup?

I have some full backups when my OS was Catalina. After I upgraded to Big Sur, I want to downgrade to Catalina again for some reason. Can I just restore from one of the backups and get Catalina again on my mac?

I have tried erasing the whole disk and reinstalling Catalina. It worked but I can not restore files from the backups any more(I got an ERROR : security agent was unable to create requested mechanism builtin:prelogin). I can only restore files when the OS is Big Sur.