Toruble setting up used Airport Express

Here's my setup. 2007 iMac running Mavericks with Airport Utility 6, hard wired to my Airport Extreme. Wi-Fi is turned OFF on my iMac. Have Apple TV hard wired to Airport Extreme. Have one A1264 that I bought brand new hooked up to a stereo connected wirelessly so I can Airplay music to via iTunes and Pandora. Had no problem setting that up over two years ago and works perfectly. Airport Utility shows Internet - Airport Extreme - Airport Express, all are green.

Now I boguht two used A1264 off eBay and am trying to add these to my network. It's not as easy as I thought it was going to be. I thought I would plug them into the wall by themselves, and they would show up in Airport Utility. They show solid Amber when first plugged in, then flashing amber, which means something's wrong. In Airport Utility, one of them shows up under the Other Wi-Fi Devices tab on the top left, and when I click on the new Airport Express, it says it can't connect. Do I have to do something to these to reset them? As suggested on some other forums, I have held in the reset button while plugging it in, and holding it in for another 8 seconds until it flashes amber rapidly, but still the same thing. What am I doing wrong here? The instructions say you must connect the Airport Express to your stereo before pluggin in, but honestly I can't see how that would even matter. I tried it but nothing. It should just connect regardless of wheteher you have a miniplug going into it.