I just want to add here. An Apple support contacted me earlier about this issue. Well, they don't really know what's happening they just contacted me to go over the issues and they took some logs on my mac. They say that a good number of mac has the same issue but they don't have any idea what causing it. But tonight I discovered something about the wireless-connected device that might be causing it.
In my setup, I have a Bluetooth keyboard (Logitech k380) and Logitech MX Master 3.
Issues appeared when I connected my mouse via Bluetooth or dongle and when I turned it off the issue disappeared. I think it is because of the software (logi Option+) but I tried connecting my other Bluetooth mouse (Generic one with no software needed just connected via Bluetooth) and I got the same issue so it's not on the software.
But I also have my keyboard connected via Bluetooth and It has no problem. The problems only appear when a wireless mouse is connected (bt or dongle). The wired mouse works perfectly fine.