After successfully installing 10.8.2 from 10.8.1 - which never worked with 96 GB (6x 16GB) on my Mac Pro, it worked perfectly with reducing the RAM to 48 GB. See posting above.
After the 10.8.2 Update I added again 16GB to a total of 64 GB - WORKED!
Then I tried it again with all 6x 16GB (3x 16 + 3x 16Gb) FAILED ... again the same frozen broken screen on 10.8.2 - whitch works PERFECTLY under 10.8.1 (!!)
Last not least I added 1x 16GB more, so right now I have 3x 16GB on Bank 1 + 2x 16GB on Bank 2 - in total 80GB and it WORKS!!!
The memory is fine ! as i changed the different RAM Modules, so it`s NOT a defective RAM Module. For me right now the solution is either to run 10.8.2 with 80GB OR 10.8.1 with 96GB.
I guess this really has to be a BUG, or has anybody else another idea WHY 96GB RAM work on 10.8.1 and (in my case max. 80 GB RAM) work only in 10.8.2?
Maybe this solves some problems for other with >64GB RAM installed when they want to upgrade from 10.8.1 to 10.8.2 .... If you can run your system with more than 80GB PLEASE let me know. Right now I have at least one spare 16GB module.
so that config is working now:
Mac Pro 2009 2x 2.66 GhZ Quad-Core
80 GB RAM - 3x 16GB + 2x 16GB
ATI RADEON HD5870 1GB
Apple RAID Card installed (with 4 drives raid 5 attached)
Apple Fibre Channel Card installed (2x 4GB Fiberchannel)
Monitor NEC PA301W
Booting from a Crucial M$ 512MB SSD