Either it is an issue just with Steam & the overlay feature, or it could be that some other third party software is interfering with the Steam overlay function. Or maybe you have a macOS setting that is quite compatible when enabling the overlay function in Steam.
I'm not a macOS software expert, so I'm not sure how much help I will be here. However, try running the third party app EtreCheck and post the report here so we can examine it for possible clues.
Depending on how much Free storage space you have on your Mac (ignore the "Available" space listed since that term is misleading), there are two things you can do to test Steam. You need to always have at least 20GB+ of Free storage space at all times for the normal operation of macOS (even more Free space is recommended).
Option one is to create a new macOS user account. Log out of the main user account & log into the new user account. Test Steam & the overlay function. If it works properly in the new macOS user account, then you know the problem is within your main user account (login item, or some customized preference/setting). Option One should not require too much space assuming Steam & your game are installed for all users on the Mac and not within just your main user account. I haven't used Steam on macOS in over 10 years & never paid attention to where it installed everything.
Option two requires having lots more Free space since it involves installing another copy of macOS beside your current installation & downloading Steam and your game for testing. For this option you would probably require having 100GB of Free storage space depending on the size of the Steam game you are testing. If you have sufficient Free storage space for Option Two, then you would need to create a new APFS volume....making sure to give it a unique name such as "Sonoma_Test". Install macOS to this new APFS volume.
If you don't have enough Free space to perform Option Two, then you could install macOS to an external USB3 SSD for testing, otherwise you would need to erase the internal SSD to perform a clean install of macOS on the internal SSD.
With Option Two, it is important not to
- sign into your AppleID or iCloud
- restore or migrate from a backup
- install any third party apps except for Steam & the game involved
Option Two will tell you if it is a hardware issue or a Steam issue with the overlay feature. Unfortunately there is no way to tell which one. If other Mac users are reporting the overlay issue, then it would be a Steam issue and only Valve will be able to resolve it (or Valve will need to get Apple to address any macOS bug associated with the issue).
Hopefully we may see something in the EtreCheck report that may give us a clue.