I don’t have an Apple Silicon Mac, so I can’t test myself for how it differs from Intel Macs.
Streams with audio in Dolby Digital or Dolby Digital Plus through the Apple TV box in Output Format: Auto setting would be in PCM Multichannel. This is as intended, and doesn’t affect quality. The stream is a bit bigger, as the audio is now uncompressed. If that isn’t working for you (i.e. not compatible), then try the setting to force output in Dolby Digital 5.1 format (fallback option). This may cause a re-encode (unless the source was DD 5.1).
As Mac is listed to support HDMI passthrough, then when that is active, it should then output Dolby Digital or Dolby Digital Plus if that is in the source, without decoding to PCM, for a direct HDMI connection between Mac and surround sound receiver.
The Denon AVR-X550BT 5.2ch doesn’t support Dolby Atmos, so Dolby Atmos formats don’t apply.
The Denon AVR-X550BT 5.2ch doesn’t support DTS:X, so DTS:X formats don’t apply.
The Denon AVR-X550BT 5.2ch does support Dolby TrueHD and DTS-HD MA (i.e. losslessly compressed surround codecs), but the Apple TV box doesn’t support outputting that using passthrough, instead possibly decoding to PCM Multichannel 48kHz (not high res) for applicable apps. My Mac (Intel) test with IINA pointed to default PCM output over HDMI from DTS-HD MA and Dolby TrueHD. I haven’t figured out if or how passthrough is possible from Mac for these formats.