I think any port on current Macs are considered hot-swappable.
However, I am personally more paranoid about FireWire ports, because I +blew one out+ on a Power Mac G5. It was the front port, so I often swapped my old iPod, video camera, and other FireWire devices on-the-fly on that port. One time, it did not work. Fortunately, fixed under AppleCare. Since then, I seem to notice more reports about FireWire ports failing (much more than similar failures for USB ports).
My precaution on my iMac is to use a self-powered FireWire hub for anything that I connect and disconnect often. I use the direct port for an external drive that I keep connected almost all the time. If I do need to connect and disconnect it, I power off the iMac. If it was a MacBook, I would probably use
sleep when I connected or disconnected a FireWire device.