Black Screen with Grey Menu Bars on certain Spaces - Big Sur

I'm having this issue after rebooting where Desktops 5 and 6 will frequently show up black with a grey menu bar once MacOS loads. For some reason, the issue only happens with desktops 5 and 6.

In order to correct this, I have to delete and recreate these desktops, which then requires me to have to reassign each application to these desktops that I had assigned to them. When I swipe up for Mission Control, the desktops appears normal with the correct wallpaper in the desktop preview strip at the top of the screen. But when on the actual desktop, it's black.

I'm running a 7,1 with a 580X and dual HDMI monitors. I have Mission Control settings set to where both displays have separate spaces. Automatically rearrange spaces is turned off. Any ideas on why this is happening? I was running this same configuration on a 5,1 and K5000 Mac Edition with Mojave and did not have this problem.