You may want to check, if your Mac has been starting up in Safe Boot Mode. This will make the Library tab vanish.
See: Photos on Catalina: If the Photos View vanishes from the sidebar, check if the Mac is in Safe Mode - Apple Community
If the Mac is not in Safe Mode, but the Photos item is still missing, try to reset the NVRAM. It helped in a similar case as pointed out by user travellerva:
Viewing 'Photos' by year on Catalina - Apple Community
How to reset the NVRAM: Reset NVRAM or PRAM on your Mac – Apple Support
If it is not a problem of Safe Boot Mode or the NVRAM, it may be a problem with your particular Mac model on Big Sur. Several users with older iMacs reported the same problem, with no fix found so far.