Yes - create the entire menu background, including any text for the buttons, in whatever app you normally use (AE, Motion, FCP, etc) and export that as MPEG2.
Take a still from the section of the footage where the button text is all in place and use that as a background layer in Photoshop.
Add a new layer and on that only add the shapes you intend to use for the button highlights - you use the background image to help align the shapes you want. Hide the background image and export the result as a .pict file. In general I make these gray scale before exporting, but you don't have to. You should end up with a single layer image which is mostly white, but has a collection of shapes such as underlines, ticks, dots or whatever (and these are in black, or gray).
Your MPEG asset becomes the menu background. You use the loop point property to identify where the button text has reached a settled point, and then add the .pict file as an overlay.
You can now draw out the button rectangles, set the targets and define the navigation between the buttons. As you draw them, the highlight shapes should appear in whatever colour scheme your version of DVDSP is using - you can adjust the colour mapping in the property inspector, just as you would for a standard menu.
The effect is that the animated background appears, the text for the menu buttons fades in to place and settles and then the menu buttons become active... this portion needs to continue for at least 30 seconds. At the end of the menu, the loop point control will define where in the background footage you return to.
DVDSP will not need to compress the resulting menu since it is already MPEG2 format, and you are not adding anything that needs to be compressed - picture highlights exist in the sub picture stream (same place as subtitles), and do not need to be composited into the menu. The only time DVDSP needs to encode the menu footage is if you add in a shape or type some text... If you intend doing that then you must only use uncompressed QT files for the background.
In your case you are including the text in the highlights and it will look pretty poor - there are only four colours in the subpicture stream, one of which needs to be white. The colour mapping doesn't anti-alias the text, which is why it is looking poor for you. You can achieve the glowing text rollovers very easily using a simple overlay.