Safari Download Location Not Working

In Safari, when I select my download location as "Ask for each download" it just automatically downloads to my Downloads folder without asking. The setting doesn't change, it just never asks. If I change the setting to a specific location, such as my Desktop, it works fine.

Any ideas how I can fix this?

I'm using an M4 Mac mini Pro and running Sequoia 15.3.1