Latency in Logic Pro (Please read before responding)

I run a recording studio and record off of an M2 Pro Mac Mini, 32GB Memory on Ventura 13.6.1 in Logic Pro 10.8.1 Rosetta. I record through a UAD apollo solo. As of about last week, I noticed I was getting completely unavoidable latency in Logic. To test out how I could fix it, opened up a blank project and sure enough I am still getting a noticeable amount of latency, probably 20-30ms. I have been recording in my studio and off this exact mac mini m2 pro for about 2 years now with no issues until now, with what I at least think is a beyond beginner level of how to run my devices but I am completely lost at how to fix this, as well are my other buddies who have more experience than me.

Before you respond, I am aware it is not due to the usual culprits like Low Latency Mode or I/O buffer size. As mentioned before, I am in a blank session with a single audio track with the input monitoring on, low latency mode enabled, and buffer size set to 128 which Logic claims will result in 13.0ms roundtrip (3.5ms output). I have tried disabling and re-enabling core audio, uninstalling the UAD console and reinstalling, changing I/O buffer speeds, recording delay adjustments, latency compensation inside of general audio, processing buffer range speeds. I went through my entire computer and cleaned all of the extra files and storage because I thought that could be what was slowing it down (it wasn't, but cool more storage). Nothing has worked besides one day when my brother was messing around and it magically fixed itself. Unfortunately just 2 days later it's back again and we have no idea how we fixed it.

My UAD console picks up no latency but it seems like as soon as it hits logic something is slowing down the signal, at least that's my theory because Logic isn't reporting the latency below I/O buffer size like it typically does if there's an issue. Adding any amount of plugins onto this of course makes the problem much, much worse and of course clients prefer to track through the plugin chains (which for me typically work fine) inside of logic rather than a blank DI, and I prefer not having to shift regions over before quantizing every single time I record.

I'm losing hope at this point. About to just trash the entire computer and buy a new one because this is so frustrating, and every single article/video I find is just another guy talking about how to put on low latency mode or mess with buffer size. Please help!