I too am having trouble playing DVDs burned with my Optiarc drive. Sorry I don't have the solution, but I can add a data point to the story about problems with DVDs burned with Optiarc.
My current hardware: mid-2007 24" iMac, built-in Optiarc DVD RW AD-5630A with firmware 1AHM / MMC-3.
My scenario: after making home movies with iMovie '08 and iDVD '08, I either burn to DVD (TDK 16x DVD-R 4.7 GB) straight from iDVD or I save as disc image from iDVD and then burn the disc image from Toast 9. In either scenario, I can play the DVD on my iMac or any other Mac or Windows computer. However, it will not play on my Xbox (original Xbox, not the 360) because it freezes when trying to mount (sometimes the Xbox will not recognize the disc as a DVD, and sometimes it will begin mounting the disc and then freeze before displaying the title menu).
Before I bought my new iMac, I had the same workflow on an older 2003 17" G4 iMac that has a Sony DVD burner, and those discs play fine on all DVD players including my Xbox. To test the hypothesis that something happened to my Xbox around the time that I upgraded to my mid-2007 24" iMac, I burned several DVDs using my older iMac, and they continue to mount and play fine on the Xbox. On both my older and newer iMac, I have Mac OS X 10.5 Leopard installed, same version of Toast 9, and I burn at the slowest speed (2x) on both. So it seems that the problem can be isolated to the Optiarc DVD burner, but I'm not sure if there are other factors at play here.
I searched the Optiarc site (www.sonynec-optiarc.com) but could not find a firmware update for the AD-5630A. However, I did find this site via Google:
Liggy's and Dee's modified NEC and Optiarc firmwares
http://liggydee.cdfreaks.com/page/en/Optiarc-AD-5630A/
Because I was not sure about the site's reputation or validity of the firmware, I have not tried it yet.