Change default app for opening music from Finder (not Music) OS 13.3.1 Ventura

In OS 13.3.1, can I change the default app for opening music from Finder? When I select a music file, hit Cmd-I for Info, then under "Open With", use the drop-down menu to select a different app, then check "Change All", I get a strange, new error message:

However, if I right-click on the file, choose Open With, and select the other app - then, once it starts playing, go back to the Cmd-I Info window and choose the default app I want, I seem to be able to do it:

This sometimes caused issues when I double-clicked on another music file in Finder: I got this error message:

However, if I go to another folder and click on a music file there, then go back, I'm able to open the file in the app I want.

I'm trying to avoid the annoying thing where Music immediately adds a song to my Library when I open it from Finder. Many times, I want to listen to an audio file but do not want to add it to my Music Library, but need more options than Space-Bar Preview offers. If I want to add a song to my Music Library, I want it to be my choice, not my laptop's.