you don't open youtube within QuickTime. you use your browser as usual for youtube. then to record the audio you choose "new audio recording" in the QuickTime>File menu after opening QuickTime.

then when i want to record music from youtube, i use QuickTime quite successfully. however, to get high quality audio you need additional software. i use the free BlackHole Virtual Audio Driver. then you can choose BlackHole as the microphone in the options.

you will also need to set up a multi-output device in your audio MIDI setup utility to be able to both hear and record the internal audio at the same time.
to set up a multi-output device:
Open Audio MIDI Setup: (found in /Applications/Utilities)
hit the '+' button in the bottom left corner and select "Create Multi Output Device"

then in the panel that appears on the right, in the Multi-Output Device tab, select "Built-in Output" AND "BlackHole 2ch". (and, if you want, any of your other connected devices.)

once you do that, (and BEFORE recording) you can choose "BlackHole 2ch" or "Multi-Output Device" in your menubar "sound options"

to both hear the speakers and record the audio at the same time, choose "Multi-Output Device". before choosing multi-out, set the internal speakers' volume to a comfortable level first. you cannot control the volume once you are in multi-out mode.
and BlackHole allows you to record the internal audio along with screen recordings of videos by choosing "new screen recording" from the QuickTime>File menu.

you can then choose "BlackHole 2ch" in the recording options instead of "Built-in Microphone."

and finally, the developers of BlackHole ask for your e-mail address where they can send the download link. but note that they haven't sent me even one spam-mail afterwards.