I played around with this a bit. Since when I decided to test it, it would not work as I said it should and I remembered it did work before.
My BT speaker does not show up in the "Choose which speakers to use" menu either. It only shows the computer, and the nearby Apple TVs
Found out if you change the Audio Playback source in iTunes Preferences->Playback, to "Direct Sound", then iTunes will use the paired BT speaker directly. You will need to restart iTunes to complete the change.
If the Audio Playback is set to "Windows Audio Session" it does not work for some reason, and instead uses the internal audio output. Whether it be headphones, wired speakers or just built in speakers.
I must have changed it at some point on my older Windows computer, and forgot about it.