Hey Louie,
I have this exact problem as well, and I for one do not have an external HD and never have had one nor have I messed with the internal drive. So for some people, maybe that was the trigger that caused it, but it certainly isn't the exclusive cause/trigger. For me the problem started immediately after a 10.10.3 upgrade and subsequent upgrades to 10.10.4 and .5 have not solved the issue. (I have a 27" late 2013 iMac).
That being said, removing the external drive should not cause any problems. Those are designed to be powered on and off and plugged in and out all the time. I would recommend using the eject procedure on the drive to stop it before yanking the USB cord but other than that you'll be fine.
Hard shutting down the computer isn't a great thing. Modern harddrives autopark so you won't damage the drive per se but things that are supposed to be written aren't and the computer each time you starts thinks there's been an error so it'll go through a disk check and take longer to boot. My (temporary) solution is to get a wired keyboard and mouse (even a cheap amazon one) as it does not suffer from the bluetooth problem. It may be slow to wake (taking up to a minute to wake up because the bluetooth controller spazzes sometimes) but it'll fix the problem.
I'm in with Apple support since the iMac is still under warranty and I've been getting bounced around a bit. The senior support guy who seemed to be helping me with it suddenly stopped responding and it turns out he is at training so I have to go back through the rigmarole a bit. The usual NVRAM resets and even a reinstall of the OS did not fix the problem so I'm being escalated. There's stuff in the console indicating an issue with the IObluetooth controller. I'll keep you posted with any updates