MacBook Air M1 via HDMI to Denon AVR - no HDR signal

I have answer from IINA support that Mac OS is controlling the HDR content not the player.

That seems plausible to me.

Given that Apple is going to focus primarily on the Apple TV 4k and the fact it works, they may not take an active interest in the IINA player or anything else for that matter. Still, I encourage you to contact them and explain all the extensive troubleshooting steps and information you gathered. From what you describe something isn't right with the way macOS is handling that encoding, and none of us are in any kind of position to determine what that might be.

Start here: Official Apple Support. You will need to be either fortunate or persistent enough to get the attention of an engineer. That will take some time, during which you are not likely to get an answer. In time, Apple may come up with a software update.

I think that's all you can do.