I am finding that iTunes and iCloud are still recognizing my new iPhone 5 as my iPhone 4s... even though I do not own it anymore. I think there is a glitch in there somewhere. I have traded out my new phone once... erased it several times... backed it up... restored it... still no luck.
I am going to keep trying, but perhaps there is some inherent old soul in the iPhone 5 that is really a 4s... therefore... your 4s won't give up its library? Have no real answer for you... just thought my problem might be in the similar realm.