Homekit invited users cannot use homekit on multiple devices

The HomeKit invitation is accepted and works fine on the first device (iPhone)

When logging into second device (iPad) on the exact same account the invitation is there but just has the spinning ball and the only thing you can do is cancel.

Is there a way to get this to work? This used to work just fine.