I think perhaps the most significant difference is the warranty (which has nothing to do with the device quality). If you buy a refurbished device from the Online Store, as I did with my MacBook Air (saved $200!), you get the same one year warranty you would get if you bought a brand new device. If Apple swaps out your device under warranty, you will have the remainder of your original warranty or 90 days, which ever is longer, if I recall correctly.
If you buy a device online, you get a brown box but all of the same accessories as you would with a new one. If your device is replaced under warranty, they hand you a naked device, no accessories, no box.
I've had one iPod, one iPhone and now, one iPad replaced under warranty. The devices were indistinguishable from new.