Logic Pro lags or glitches with RME Fireface UFX II on M1 Mac

I have for some time now had a very frustrating problem with logic when connected to my Fireface UFX II.

Whenever im connected to the interface (meaning its configured as either Input device or output device or both) in, then when the project gets to a specific size (ie several software tracks and some medium heavy plugins) the playback sometimes dont want to play. It kinda tries, but keeps glitching or lagging, so it never plays more than a few ms of audio and the tries again. This is also visible as the yellow microphone icon in the mac menubar blinks at the same time.

This could ofc feel like a computer cpu-problem. BUT when i plug a headphone jack in and deselects the UFX in logic, it plays perfectly fine with no lagging or cpu-problems at all...

This is sooo frustrating because I then cant record or playback sessions that i have worked on more than an hour or two.

Has anyone had similiar experiences?

I have tried all combinations of the RME kernel and driverkit driers during the year or so, and have been in contact with RME support, who just wants me to recreate the problem i other daws - which i cant because they apparently handle this problem in other ways. So it doesnt do it with Albeton etc... And then they just give up and cant help. Or stops answering...

I have a Macbook pro M1 16" 2021. Running Sequoia and Logic 12.0.1 (but has had the problem for loong time).

and the newest driver (tried both kernel and driverkit).