Unlike older Mac Pros that required matched pairs, the 2009 and later models of Mac Pro 65lb tower have four slots on each side (each processor module), and they operate somewhat independently. The first three slots are independent, the fourth slot is shared with the third slot.
1) If you install one module on each side, it will run slower because it cannot interleave.
2) If you install two on each side, it will run about five percent slower than optimum. This does not make a measurable difference.
3) Three modules on each side runs at the fastest memory speed.
4) Four modules on each side runs five to ten percent slower, because slots three and four are shared. In some cases, it may run one step down from its fastest speed. The slowdown is memory speed is usually more than compensated by faster overall operation from having adequate memory.
What you want to avoid is filling most of the slots now AND needing more later -- because you will have to throw modules away in that case.
So fill about half the slots with the total amount of memory you want. (Example: If you want 24GB, buy four 8GB and install those, two on each side. If some of your older modules will work with them, great. That way you still have lots of room to grow.)
Executive summary: Don't buy lots of 4GB modules, the 8GB modules are pretty affordable. If you are short of cash, buy some now and add more later.