WindowServer high memory usage

I have just started switching from Windows to MacOS with my MacBook Pro 14inch, M1 Pro, 16GB. I had a problem with memory usage of WindowServer which was it always comes up to 5-6GB even after I unplugged it from my external monitor.

I have searched around internet for this issue and find the normal memory usage of WindowServer is around 1-2GB so how can I manage my WindowServer memory usage around that number. Thanks

The image bellow is my memory usage of WindowServer after I unplugged from external monitor