Roundtrip Latency won't go below 30ms on M1 Mac no matter what

I have a M1 MacBook Air with Logic Pro 11. I sometimes use my Steinberg CI1 audio interface, and sometimes just headphones plugged directly into the Mac.

Recently, started using Logic for the first time in a few months and I saw that the roundtrip latency was SUPER high. Using built in speakers and mic at 32 samples (the minimum) I get 33.8ms roundtrip latency, and with the CI1 interface I get 33.2ms.That's completely unusable in terms of recording and monitoring yourself. I don't remember exactly what it used to be, but it was never this bad (pretty sure I could get the roundtrip latency down around 12-15ms before, but with some crackling depending on how many plugins were in the project, IIRC).

This is with a completely blank project (just 1 audio track with nothing in it), so plugins aren't the issue here. I know the M1 isn't the newest, but it should be able to run a completely blank Logic project with far less latency than that.

Not sure what happened, the latency never used to be this bad. I had a backup of Logic Pro X on my mac so I loaded that up, but the latency was just as bad as it is on the current Logic Pro 11 build. Not sure what to try next.