First spaces merge when disconnecting from second display

I've noticed that when I disconnect my laptop from my second display the first Space of each display will merge. For example, if my monitor has "Desktop 1" and "Desktop 2" on it and my laptop has "Desktop 3" on it. When I disconnect my display "Desktop 1" and "Desktop 3" will merge and "Desktop 2" will be added to the laptop. So now instead of 3 Spaces I will now have 2, where 1 of them is a merge from the first 2 on each display. It's always the first 2 Spaces that merge. Even if each screen has 5 Desktops the laptop will have 9 Spaces after it's disconnected. Is there a way to have it so all of the Spaces are on the laptop when I disconnect my display instead of merging?

Thanks!