Polka Salad,
Here's how to get your visualizer to change sizes and not start in full screen mode.
step 1: open iTunes.
step 2: go to edit and then preferences (or ctrl + ,)
step 3: go to the advanced tab, (when your in the advanced tab make sure your in the general settings tab.)
step 4: at the bottom of the general settings page you should see a drop-down menu for the visualizer size (you can chose from small, medium, and large). Below that drop-down menu there should be a check box for display visualizer full screen.
step 5: uncheck that check box
step 6: save your preferences by clicking ok.
that should solve your problem now when you open the visualizer it should open in the iTunes window and then you can use (Ctrl + F) to launch the visualize full screen.
Hope this helps,
DJFX