I don't use Handbrake, but looking at the decoding dialog on their site, I can make a few suggestions:
Select Video Codec: H.264 (if you have real 264 use it... I don't trust x264 [although videos I've seen lately have improved.)
Video Quality: Select Average Bitrate and set that to no less than 15000 (for 480) — higher is better — max about 50000. A little overkill is NOT a bad thing. Select 2-pass encoding!!
DO NOT ever use Fast Decode.
Never set a variable frame rate (VFR).
Set everything you can to BEST (and H.264 level to its highest setting)
You should also match the output Framerate to the original framerate (typically for DVD it will be 29.97i or p -- if p - you're lucky.)
If you can match the resolution (720x480) it will be better and if you can set an anamorphic flag for widescreen, then do so (otherwise, for widescreen try a pixel resolution of 854x480 with square pixels).
Do not deinterlace, Decomb, Denoise, Deblock or anything else. Deal with all of these in FCPX. ONLY allow Handbrake to make the transcode.
No matter what, you're going to have "issues" going from DVD to a 50in HD (if that's all it is!) display. FCPX does a pretty decent job upscaling if it has the bandwidth.
