I came to learn that it was actually just bugged... I rebooted into the system and it works as intended now.
I tried to reproduce it but when I run Minecraft, the overlay works and when I close it, my iTerm is properly triggered without any game overlays showing up.
But before writing it, I really checked that when I start a game, I have a game mode icon in the menu bar and when I close a game it disappears. But for some reason, the overlay shortcut still was triggered even when no game and no game mode icon were present...
Apple, fix this please, it really shouldn't be a thing :)