Another possibility is that these stubborn songs were "iCloud downloads" ... if you have Show All turned on in your Music settings, you'll see a cloud icon next to all purchases that are not on your device, as AmishCake mentions out above. If you tap one of these clouds the item will download to the device, and in my experience they become difficult to remove.
See if it is just a setting change needed as pointed out by AmishCake, but if not then here are some things you can try:
1- Try swiping across Artist names, Album names, or Songs, from right to left, then tap the Delete button. If you have a lot to delete, then delete by Artist. Deleting things this way may not always work for iCloud downloads though.
2- Go to Settings/ General/ Usage/ Manage Storage/ Music, and then swipe from right to left across "All Songs" and tap Delete. I wouldn't do this option if you still have songs synced to the device, as this tends to cause quirky things to happen playback and artwork, requiring reloading your music.
3- The brute-force method I've had to resort to is to change the device to "Manually Manage Music and Video", remove all the songs that I can through iTunes navigating through the iPhone contents, do a backup of the iPhone to iTunes, restore the iPhone, then reload the backup. If all songs are gone (they should be), then you can go back to auto-syncing just the songs you want.