Manually managing music does not work

I am having a hard time managing my music and syncing too much to my phone. All software is updated and i've gone through the "manually manage music and videos" tutorial many times. My problem is that my computer is trying to copy two of each song from my library to my phone. Even that, it o\is only allowing me to listen to a fraction of that list and most of the music is grayed out on the phone its self, and every time i try to do something it tries to sync taking hours and by the time it gets close, my phone disconnects. I want to be able to plug my phone in and put songs on and off from the computer and manager everything without it always trying too sync EVERYTHING from my computer. All help is largely appreciated as this has been an ongoing issue for some time I also do not use apple music, just old school buying from itunes