I have to use AirPods to listen to music on my Apple Watch SE?

Because a phone call tends to be relatively short, and the speaker is statistically only making sounds for half of the call duration, assuming the other party lets you get a word in edge wise.

Music is non-stop for hours. Your Apple Watch battery is very small, and if you are streaming music, that also means you are running one of the radios (Bluetooth, WiFi, or Cellular). This kind of activity will quickly drain your tiny Apple Watch battery.

Here is the battery estimates for the Apple Watch battery with different usage

https://www.apple.com/watch/battery/

• Talk - Up to 1.5 hours connected to LTE
• Downloaded music Audio Playback Up to 11 hours playback from Apple Watch storage going to Bluetooth headphones
• Streaming Audio playback Up to 8 hours streaming playlist with LTE going to Bluetooth headphones

So yes you can use the Apple Watch speaker while talking on a phone call, but if your phone call lasts an hour and a half, your battery is dead. Stream music via Bluetooth headphones gives you 11 hours, but if you were using the speaker, it would be the same as a phone call, and your Apple Watch would be dead in an hour and a half.

You'll find that punctuation makes it possible for others to understand what you're asking.