Homepod Mini - Setup Failed (-5320)

Bought 2 Homepod Mini's to add to my house network today. Got home and tried to set them up. Both Homepods were stuck at configuring. One asks me to input my Apple ID password. I reset the password after it failed several times and asked again, and it continued to say it failed. I plugged them into my Mac to factory reset and install the newest firmware and now I'm getting the above error code when the Homepod asks me to sign in during setup. Any ideas? I've restarted my iPhone and Macbook Air, my phone is on the latest iOS. I have another Homepod Mini already configured on the Home and it works perfectly fine. I've also tried adding a "Test Home" and "Test Room" to try to see if they would work there. This has been going on for several hours and is starting to get frustrating. Seems like a lot of people have trouble adding extra Homepods to their Home.