Hello jonpais,
For best viewing experience, in the situation you describe, it can be a bit of a personal preference. But, referencing the article that linked you, there is some information on both modes that may help you make that determination.
Use reference modes with your Apple display - Apple Support
Pro Display XDR (P3-1600 nits)
On Pro Display XDR, use this mode for general home and office environments. This mode is based on the wide color P3 primaries used by Apple displays and includes Extreme Dynamic Range (XDR) support up to 1600 nits (peak).
HDR Video (P3-ST 2084)
Use this mode on MacBook Pro or Pro Display XDR for 4K or ultra high-definition video production workflows up to 1000 nits (full-screen sustained) using the wide color P3 primaries and the high-dynamic-range SMPTE ST-2084 EOTF. This mode is designed for controlled viewing environments set up per ITU-R BT.2100.
Typically, users don't need more nits (brightness) in a darker room, that's saved for when the screen is already in a bright environment. Also, the (P3-ST 2084) mode is best for HDR productions, which is what you might prefer when watching movies. You may need to test both modes to see what you prefer, but most likely you'll want the P3-ST 2084 mode for your purposes.
Enjoy and take care.