Okay, so I had this problem too, and I spent a while trying to fix it and here's where I came up with. *Side note: there are, like, three different music tabs that mean all different things. There is Library Music (in the library column), iPhone Music (right below the settings tab when you click on your device), and Music on your device (under the on your device tab).
1) Connect your device to your computer and start iTunes.
2) Make sure that you have all the music that you want to add/take off of your device checkmarked. I personally find this easier if you change the view to be organized by songs instead of albums or artists.
3) Then click on the icon that looks like your iPhone/iPod and it will take you to the summary of your device.
4) Go to the iPhone music and make sure the 'Sync Music' checkbox is clicked.
5) Click on the circle button that says 'Entire music Library'.
6) Sync your iPhone.
I hope this works for anyone having the same stupid problem.