Use the guides over at http://www.ifixit.com/Device/iPod to walk you through the process. The site also offers the replacement parts, although you can find them much cheaper at other online sites and resellers.
I was able to find an all-in-one kit specific to my older ipod at my local electronic store (Micro Center.)
The kit cost $20 (a couple years ago) and included all the tools I'd need to take apart the unit, the new battery, and a CD-ROM with a step-by-step video. I think it even included a sleeve to mail in the old battery for recycling.
It was easy, cheap, and kinda fun! My ipod works great to this day. Highly recommend it!
One thing I wish I had asked before doing this: Do they sell memory upgrade kits for older ipods? If so, it would have been best to change out that component while I had the unit open. Since they're not really meant to be opened by the user, I don't think I'll dare open it up a second time. Just some food for thought.