Aside from Drew's excellent links, the process is this:
1) create your menu background with all of the graphics in a single layer - make it look like you want the menu to be when no button is selected. Add all text, etc for the buttons. Flatten it and save as a pict file, 24bit colour.
2) add a layer to this photoshop image and just add the highlights that you want - a tick, arrow, underline, etc. Hide the background layer and convert this new layer to grayscale (you will need to flatten the image), then save it as a .pict file, 4 bit max - 2 bit could do it, too (You might now want to use the history tool to undo certain steps and save the file as a psd file so that you can make changes if you need to).
3) import both .pict files into DVDSP and set one as the background and the other as the overlay into your menu. In the menu editor, click and drag the mouse over the places where the buttons need to be. Click in one of those button hotspots and look in the property inspector under the 'color' tab. Use the sliders there to adjust what you want the highlights to be in each button state - remember that you are using a simple overlay in gray scale.