Having realtime playing issues with Ivory II in Logic Pro

I am using a brand new MacBook Pro 2.9GHz Intel Core i7 with 16GB 2133 memory.

When playing the Synthogy Ivory II stand alone software I can play without any latency issues or audio hiccups with a setting of only 64 samples. The Ivory samples are loaded on the internal SSD.

However when playing Ivory as a plugin within Logic Pro X v10.4.0 I still have issues even with a setting of 128 samples. I've tried variations of all the other settings to no avail.

I am using a Universal Audio Twin interface and the only plugins set in Logic are UAD plugins and they have been deactivated. I have deleted all other instrument tracks but for one MIDI instrument and one vocal track which is not armed and I still can't play Ivory with out audio interruption. And of course the latency is really annoying.

Does anyone have any suggestions on how to make setup this work at 64 samples.