Macbook Air M1 2020 - Multiport Adapter Issue

I have a USB-C Digital AV Multiport Adapter for my 2020 Macbook Air M1 but the HDMI keeps disconnecting and reconnecting to my external monitor.

It will be working fine, then the monitor will go black and everything I had on that screen moves to the Macbook screen for anywhere from 1 second to almost a full minute. Sometimes it will happen maybe once an hour and other times it will happen every few minutes and nothing I have tried seems to have any effect. No amount of moving the adaptor or cable in different positions seems to make it better or worse and I have tried multiple HDMI cables so I know that is not the issue.

Has anyone else had this issue and/or know how to fix it or at least make it happen less frequently as my only other option is to replace the adaptor which I want to avoid if possible.