Since it's a startup issue, you could try resetting the PMU (power management). The procedure for a slot-loader iMac is shown here.
http://mrjcd.com/junk/PMU.jpg
(As stated, be sure to press the button just ONE TIME between startup attempts.)
If that does not help, you might try getting a self-powered USB hub, if you don't have one already. Then connect all of your USB peripherals to the hub. By self-powered, I mean it has its own power supply. Since the hub is already powered, maybe it won't cause this issue at startup because it would not draw power from the iMac's port, and the connected devices will get power from the hub instead of the iMac's USB port. Current USB 2.0 hubs should work with USB 1.1.