I've had the very problem starting with a 2015 iMac. We've still got that model and two 2020s now and none of the card slots have ever worked satisfactory. For that reason, and others, I've abandoned cards in Macs as well as thumb drives for anything other than bootable MacOS installers.
A while back I bought a pair of less expensive SSDs (PNY by brand) at $28US each at the time. As far as SSDs are concerned, the speeds are a little slower ... 380 MBps to 420 MBps (USB-A) read speeds but reliability has been fine. Even at the slower speeds they run circles around any SD card or flash drive I've ever owned.
My wife is a photographer and the cards in her cameras work as expected. But even then, she either send the files wirelessly to her iMac or cable connects the camera.