Same here, and ironically I was on a call to the support bods about about another Big Sur issue. The agent told me it was a 'new feature' of Big Sur to allow a seamless use of the airpods between all your devices and it will automatically switch them from one device to the other. The agent said there was no option to turn this off, and the only way to deal with it was to disconnect them from Mac by removing the airpods from the bluetooth list which then removes them from ALL devices!
I was then balancing my phone under my chin, trying to speak with him, and then asking him to wait whilst I re-added my airpods to my phone so I could carry on my 1.5hr long phonecall! During said 1.5 hours, this happened four times. Just **** stupid and annoying.