When you get that error screen after you do a Reset on the iPod, the cause is often a faulty hard drive. Putting it into Disk Mode is the right thing to try. iTunes should already be running when you connect the iPod. However, if iTunes does not detect the iPod, the iPod's hard drive may not be working at all, so there is nothing to connect for iTunes.
In Disk Mode, what does the iPod's screen say (or have on it) immediately after connecting the iPod to the Mac (with iTunes running)?
You should confirm the USB docking cable is good. Try a difference cable (if you have one), or use the current cable with another Apple device, to confirm it works properly.
Also, connect the iPod (in Disk Mode) with iTunes NOT running. Run Disk Utility. Does the iPod appear in Disk Utility's sidebar?