Reading the replies, it appears that the solutions and causes are inconsistent.
I have frequently had problems over the years running 3 screens off two cards. Usually just settings and order of monitors being reset.
Perhaps the issue lies with the plist / preferences file for displays. In the past I have had to manually edit and alter permissions.
From memory, I think I manually edited the settings I wanted, then set read only permissionson the relevant plist to stop the system changing it back. This may be a red herring for a solution, but given it has been an issue with each upgrade. .. the problem may still reside in this area.
The problems seem to be how the system triggers calls to the correct display card. Either multiple spaces or the cursor near a boundary that might be considered access to the next monitor. If the system is getting confused about where and what display it should be showing, it panics.
Edit* Do the panics still happen if one desktop is mirrored to all screens? Rather than disabling cards or monitors.