Using the High Quality 90-min preset in Compressor is the best you can do with the Final Cut Studio apps. Then take that into DVD SP to author.
Know that this is making an SD DVD...so if you have an HD film, it's no longer in HD. That might be part of the quality issue you are seeing.
#42 - Quick and dirty way to author a DVD
Shane's Stock Answer #42 - David Roth Weiss' Secret Quick and Dirty Way to Author a DVD:
The absolute simplest way to make a DVD using FCP and DVDSP is as follows:
1. Export a QT movie, either a reference file or self contained using current settings.
2. Open DVDSP, select the "graphical" tab and you will see two little monitors, one blue, one green.
3. Select the left blue one and hit delete.
4. Now, select the green one, right click on it amd select the top option "first play".
5. Now drag your QT from the broswer and drop it on top of the green monitor.
6. Now, for a DVD from an HD source, look to the right side and select the "general tab" in the track editor, and see the Display Mode, and select "16:9 pan-scan."
7. Hit the little black and yellow burn icon at the top of the page and put a a DVD in when prompted. DVDSP will encode and burn your new DVD.
NOW...if you want a GOOD LOOKING DVD, instead of taking your REF movie into DVD SP, instead take it into Compressor and choose the BEST QUALITY ENCODE (2 pass VBR) that matches your show timing. Then take THAT result into DVD SP and follow the rest of the steps. Except you can choose "16:9 LETTERBOX" instead of PAN & SCAN if you want to see the entire image.