Place the iOS device in Recovery Mode and then connect to your computer and restore via iTunes. The iPod will be erased.
If recovery mode does not work try DFU mode.
For how to restore:
To restore from backup see:
If you restore from iCloud backup the apps will be automatically downloaded. If you restore from iTunes backup the apps and music have to be in the iTunes library since synced media like apps and music are not included in the backup of the iOS device that iTunes makes.
You can redownload iTunes purchases by:
Note the 2G iPod can only go as high as 4.2.1
Many developers when they updated their apps to support iOS 5 and 6 dropped compatibility for iOS 4.2.1 and earlier.
Your only way is if you have the version that was compatible on your computer such as in the recycle bin/trash or in a computer file backup. In that case delete the current version that is in your iTunes App library and replace it with the older version. Then sync.