Safari 16 changes Space when Cmd-Tabbing while playing a fullscreen video

If you fullscreen a YouTube video while having two screens and Cmd-Tab between apps, whenever you focus on Safari or click the Safari icon in the dock it automatically shifts away from the Space where the video is playing and goes back to the tab saying "Click to exit full screen". This started happening after updating to Safari 16 and I can't find a setting to disable this behavior. Is this a bug or is there a way to rollback to the previous behavior (where Safari would just stay in the correct Space)?