Macbook Pro M1 Max Logic Pro Midi Latency

I have a 16 " M1 Max with 32 Gb Ram

Im getting slight latency when using Logics Instruments both with my External Midi controller and with the built in Qwerty keyboard. Enough to make playing anything rhythmical very difficult.

Using a USB hub or direct usb cable to computer makes no difference with the Controller, but as its also happening with the Qwerty keyboard then it must be an internal issue.

Ive tried Lowest Buffer settings, higher sample rates etc. All combinations.

Strange thing is enabling Low Latency mode and even removing all effects plug ins makes no difference to having them on.

At Buffer Size 32 and sample rate of 96Khz, I'm getting resulting latency 4.1ms( roundtrip) and 2.2 (Output) which is the best i can get it which is still pretty unplayable with anything half rhythmical

At Buffersize 32 and sample rate 44.1Khz, this jumps to 6.2msRoundtrip and 3.8msOutput

Whats going on? is it something really simple I'm missing?

Thanks very much in advance!!