"Good news, everyone!"
After weeks of struggle and hate towards Apple, I managed to fix this issue on my M1 Air. And it wasn't Apple's fault.
I was so angry that I contemplated buying a Linux laptop.
I ditched my Bluetooth headphones and worked on Mac speakers for some time.
Fortunately, after that transition, Zoom, being as stupid an app as it is, still keeps changing my audio device to "Apowersoft Audio Device," leftover from the Apowersoft Audio Recorder I installed and removed on my Intel Mac Pro a few years ago.
Digging deeper, I found it in the "Output & Input" section in Preferences >Sound and further in System Information > Applications listed as Kind: Intel and in System Information > Disabled Software.
If you can replicate the above (It can be a different device, anything that's not your mikes and speakers), you are on a path to success.
I will describe how it works with Apowersoft Audio Recorder, but your case can be similar.
What you need to do is find ALL files with "Apowersoft" and delete them (also remove them from the trash).
The Finder can only find some of them. You must use the terminal or this soft -> https://apps.tempel.org/FindAnyFile/ (if you can't use a terminal, that app helps you, and you earn more than 6$ an hour, you should consider buying a license. It saves you much more time ;)
If you use the "FindAnyFile" app, you will be required to grant it admin permission, and after it deletes all it can, a few files will still be shown. Thats OK. You can restart your Mac, and you are good to go.
Please let me know if it works on your devices. Good luck.