How can I choose USB PnP Audio Device over HDMI/Dock Connector?

I am using a Lightning to HDMI+USB 3+RJ45+Charging adapter on my iPhone 13 Pro Max (iOS 15.3). Using a USB sound card on that adapter for audio output. It works fine. But when I connect the HDMI cable, it automatically switches the audio output to HDMI/Dock connector. I didn't find any option to change it to USB PnP Audio Device. Is there any way to get audio output via USB PnP Audio Device while using HDMI for screen mirroring? I don't need sound via HDMI. Thanks in advance.