Keynote Remote for iOS: Can use iPhone as remote, can't use iPad as remote

I have my iPhone and iPad set up with my phone acting as the remote control for Keynote. It was pretty straightforward.

Then I decided that I'd prefer to present from my iPhone and use the iPad as the remote. I checked to see if this is possible, and YES either device can be used to remote-control a Keynote presentation on the other.

However when trying to use the iPad as a remote, it can't find the iPhone. I have the iPhone with a presentation going, but the iPad is stuck in device search mode.

I temporarily switched back to using the iPad as the presentation device and the iPhone as the remote controller, and that still works fine. But it doesn't work the other way around. Why would it work in only one direction? Any ideas?

Both devices are on iOS 17.5.1 and the latest version of Keynote.