Make sure on the second drive that -
• the System Folder is at root level of the jard drive. Meaning, that it is visible in the initial window seen when opening the icon of that drive.
• the System Folder is "blessed". This status is indicated by the System Folder displaying the Mac OS glyph superimposed on the folder icon.
• the second drive is not larger than 200GB, or, if partitioned, that the volume being used does not exceed 200GB.
Unplugging the first drive won't work, unless you also reset the second drive's jumpers to Master (or Single if it is a WD brand drive and offers that setting) and disconnect the first drive from the IDE/ATA ribbon cable.
If you did not reset the second drive's jumpers, they may be set wrong. With two drives on a pre-QS G4, use the Master and Slave jumper settings when there are two drives. Note that if the original drive is a WD brand, it may have been set to Single; in that case, it needs to be reset to Master when adding a second drive.