Allan has it right for that Mac.
The difficulty is that different large-sized modules have a habit of not playing nice with smaller modules. Also, slot 3 is shared with slot 4 on each side, so leave those empty until you know every module works independently.
I suggest you install all your new modules first, balanced across the two sides. "Install in pairs" is not used in this model. Lowest numbered slots in each group (front left and right rear) first. so that is new modules in 1, 2 and 5, 6.
¿Are all modules working that way?
-Yes, add an old module to slot 3 and 7 and re-test
-No, find non-working modules and test with ONLY those modules
¿Old and new working in 3 slots on each side?
-Yes, try adding in the fourth on each side
-No, try old modules swapped around with each other, but you may have to leave the last slot empty on each side.