How to resolve 'Success' window from captive.apple.com when connecting to a Meraki network on Mac?

I have recently seen an issue with Macs on my corporate network - it is a Meraki wifi using WPA2 only and configured with no splash page. The profile is pushed using Jamf

When connecting, the Mac opens a window titled 'Join network' and all it says is 'Success' in the window, with a button marked 'Done' in the bottom left. It says captive.apple.com on the bottom of the window.

It appears to only do this the first time it connects, and it prevents me from using the network to register a new macbook (it prompts to say there's no internet connection because it won't open the captive.apple.com window during setup)

As far as I can tell, this has only been happening with more recent versions of Sequoia (15.3 and up)

Has anyone else experienced this with Meraki or any other network?

[Re-Titled by Moderator]