I've had this problem for ages - some grey songs on my iPhone, visible in the iPhone's Music using iTunes on the PC. They do not show up in the Music player on the phone itself. On the PC iTunes, they have a little 'resync' icon next to them, but you can't click on this, and you can't delete the songs. Even if you try to delete every song on the iPhone (using the iPhone's Music player, or using PC iTunes), they stay.
In my case, all of the tracks with this problem were purchases I had made *on the phone* before I did a full restore, which was when the problem arose. (But all the same, it didn't happen with all such tracks, only with some of them.)
I finally fixed it, but not in a very orthodox way. I basically deleted ~/Media/Purchases, ~/Media/iTunes_Control/Artwork, ~/Media/iTunes_Control/iTunes/Artwork, and ~/Media/iTunes_Control/iTunes/MediaLibrary.sqlitedb*.
Before doing this I stopped the Music player on the iPhone. After doing this, I resynced the iPhone with iTunes. This process put all the music back, with no greys and no duplicates, and it even (finally!) got the information correct on my phone again about which purchases were on the phone and which weren't.
If you don't know how to do this, this isn't the solution for you. Even if you do, it probably isn't the solution for you.
Back up your phone first, you might have to do a full restore. Caveat emptor (literally) - I hadn't made any purchases on my iPhone except songs and apps, and I made **** sure these were all in iTunes before I did anything.
I have an iPhone 4, iOS 5.1 - I have no idea if the file system is the same on other devices (and I am pretty sure it isn't, below iOS 5). If it isn't, this (clearly) won't work, and is even more likely to necessitate a full restore.
Mike