I'm having a problem very similar to this but none of the solutions mentioned here work for me. I'm using a Samsung HP-S4253 (plasma, 720p) monitor, using the VGA connection. I keep getting the "Not Supported Mode" when I power up my Mac Mini and I can't even start in Safe Mode!
I tried it twice. Once holding down the shift just until I saw the grey apple and "turning gear". It took much longer to get through that screen than ususal but eventually went to the NSM once it got through it. The second time I tried it, I did what the poster above suggested and held it throughout that screen, waiting for a "booting in safe mode" message but never got it, it went to the NSM screen.
There's a long history for this problem. The TV used to work. I'll start from the beginning.
1. I assumed that since the Mac Mini doesn't have built-in support for 768x1360, and since it automatically picked 768x1024 (and stretched out the screen) that I wouldn't be able to use the full resolution for display. So I purchased and installed DisplayConfigX with settings that I thought would work with my TV. When I did, and restarted the Mac, it got through the grey apple screen and then went to NSM (for the first time ever).
2. I followed DisplayConfigX instructions to remove the DisplayConfigX settings by accessing them in Single User Mode (aka Unix Mode) on the Mac. I restarted the Mac and was able to get the original/default 768x1024 resolution.
3. I re-installed DisplayConfigX, this time with settings I thought would work, as well as the default 768x1024 (@60Hz) resolution.
4. I restarted and it looked OK but then I navigated to system settings and clicked on Displays. I think I clicked too many times and accidentally selected a resolution that doesn't work, at which time it immediately went to the NSM.
5. I went back to Single User Mode and deleted the DisplayConfigX settings again, but this time it didn't work.
6. I took my Mac Mini to work to connect it to my LCD at work and set the display settings to 768x1024 @60hz. I brought it home, booted up the Mac
then powered up my TV but still got NSM.
7. I reset the PRAM (
http://docs.info.apple.com/article.html?artnum=2238).
8. I found a thread (
http://forums.macosxhints.com/showthread.php?t=8235) about how to edit the com.apple.windowserver.plist file (again under Single User Mode) but when I opened it in Pico, I couldn't find any of the display settings. It had the parameters included but no numbers (plus lots of garbage characters).
9. I deleted the com.apple.windowserver.plist file and restarted the Mac but it went right back to NSM and it didn't re-create the com.apple.windowserver.plist file as I expected. Instead, when I navigated there again in Single User Mode, it had a file named #com.apple.windowserver.plist#, which I couldn't delete or edit.
PLEASE HELP!!!