Recent Intel Mac Pro models had electrical (loading) issues on the DIMMs, and had to resort to Registered DIMMs, where an additional output Register (and its accompanying delays) are inserted into the circuits, slowing access times.
When the circuit traces are on an ordinary printed circuit card, wider busses to RAM memory go from merely difficult to unworkable, because they become big silver highways and have issues with slightly different propagation delays between signals, and they are harder to manufacture. If the total length of the traces gets close to half a foot, additional beefier driver circuits are needed. If you need to cross a card-edge connector, or support anybody's RAM chips, those drivers need to be even bigger. Apple's designs avoid all of that. Yup, not user upgradeable.
When it comes to connecting to memory, when you know your RAM devices are very close by (on the same carrier card) you can control for electrical issues like capacitance and propagation delays. You provide drivers on the CPU/GPU chip already tuned for the RAM you know will be connected, you can save many delays and inconsistencies.
Because they were able to get the initial M1 speeds up equal to Display RAM speeds, there is no need for separate, power-hungry Display RAM and no copying back and forth, and no special access to only Display RAM. Everything can be simpler, and that means faster.