A bit of new info from here: View and remove associated devices in iTunes - Apple Support
"If you can't click Remove (the button is grayed out), the device that you're trying to remove has connected to your Apple ID within the past 30 days. Sign out of the iTunes Store on that device, then try to remove it."
So this partly contradicts the marked "answer" above. Rogue devices will NOT automatically fall off, but will become eligible to be removed after 30 days. But if someone has your rogue device and is actually using it it will never time out.
In my case I had 8 devices grayed out and two not. I removed those two. But this only helps a little. Some of the others are duplicates of the same device. Since that device keeps connecting I suspect that I will never get past the thirty days for either copy of the device.
So it's still broken, but makes a little more sense.
I should also note that the suggestion of going to https://appleid.apple.com/ and removing the devices doesn't work for me. To remove a device from that list you click on it and select "Remove iPhone". See Check your Apple ID device list to see where you're signed in - Apple Support. But this appears to only be relevant to iCloud, not to iTunes or Apple Music. I brought the list from nine devices down to two, but then when I checked iTunes it still had eight devices and all were grayed out.