Setup Apple TV Homekit to iPad "Setup Failed (-6712)"

I am trying to set up HomeKit and my homepod and running into multiple setup issues. My current one is on my apple tv, updated to 14.3, my iPhone at 14.3, my iPad at 14.3. I have to log out of iCloud and when I log back in I get a message to setup my home. The apple tv tells me to have an iPad or iPhone close by, I get a message to set up and then on the iPhone I get the "Setup Failed (-6712)" doing it all over again and putting my iPad next to the apple tv, I get asked to enter the 4 digit code on the screen and then I get "Setup Failed (-6712)".