Windowserver taking 3,20GB memory when connected to dual 4K Displays + 12.9 iPad via sidecar using 14" MacBook Pro 10+16+16GB+1TB

Hi,

I have a couple of issues with my MacBook Pro 14" M1 Pro 10CPU + 16GPU with 16GB of RAM and 1TB.

Whenever I have my dual 4K Huawei Mateview displays connected and using Sidecar with my 12th Gen iPad Pro I get around 3,20GB memory on the process Windowserver.

Additionally whenever I disconnect any of these displays my computer starts to aggressively swap around 2GB and causing yellow/red lines across the activity monitor.

Print screens attached

Please advice on how to fix this issue because as you can see I have created some 2GB swapping just by turning off one of their displays.

I am running the latest version on all software.