Update.
I thought QuickTime recordings of System Preference resolution choices shouldn't be able to record at twice the selection above the maximum Retina resolution, but the MacBook Pro seems to do just that. My Retina iMac does not.
I tested various choices with my Macs
iMac Mid 2015 27" (Retina 5120 x 2880) 16:9 aspect ratio 0.5625
System Preference Selections QuickTime Records screen at
3200 x 1800 Most Space 4096 x 2304 1.28x
2880 x 1620 More Space 4096 x 2304 1.42x
2560 x 1440 Default 4096 x 2304 1.60x
2048 x 1152 Larger Text 4096 x 2304 2x
1600 x 900 Largest Text 3200 x 1800 2x
MacBook Pro 16" 2019 (Retina, 3072 x 1920) 16:10 aspect ratio 0.625
System Preference Selections QuickTime Records screen at
2048 x 1280 More Space 3686 x 2304 1.8x above screen resolution
1792 x 1120 Default 3584 x 2240 2x above screen resolution
1536 x 960 Larger Text 3072 x 1920 2x
1344 x 840 Much Larger 2688 x 1680 2x
1152 x 720 Largest Text 2304 x 1440 2x
iMac Late 2013 27" (2560 x 1440) 16:9 aspect ratio 0.5625
Recorded everything at selected resolution (non Retina display)
System Preference Selections QuickTime Records screen at
2560 x 1440 Default 2560 x 1440 1x
2048 x 1152 2048 x 1152 1x
1600 x 900 1600 x 900 1x
1280 x 720 1280 x 720 1x
examples
iMac 2015 27" Retina QuickTime recording
MacBook Pro 16" Retina QuickTime recording
So test on your system.
ivan