iCloud Music Library & iTunes sync

I have Apple Music but also have a large iTunes collection of music I've downloaded myself over the years without any input from Apple. This includes tracks that aren't available on Apple Music. Whenever I want to put a song on my iPhone that isn't available on AM I of course have to disable iCloud Music Library so I can put the song on my phone, and then I re-enable iML. My main issue is that every time this happens, a large portion of the music already on my phone (from my computer files, not AM) appear in my Music app with the 'Download' icon next to them, even though they should be locally stored on the phone as I transferred them using iTunes originally.

How do I stop this happening? It's really driving me insane and I don't want to have to re-download half my library so I can listen to it offline when it should be locally stored in the first place.