I had a setup that worked perfecly for a couple of years, with a mac mini running hard wired to a router that also had wireless, with the express linked by wireles.
Then after some mix of OS, iTunes and express firmware updates it started having this intermittent problem.
Then, I tried everything. Using all ethernet hardwire, upgrading router to latest time capsule, getting the latest airport express, nothign made any difference. I could airplay to the ATV no problem-- but the audio is degraded because it resamples, so that was not ideal.
Finally after returning the new express, trying hardwire again, ipv6 off etc etc, I tried the firmware downgrade on the express that does the audio to my high end system. From 7.6.4 -> 7.6.1. (in the new airport utility, option click the version number in the aiport's info panel).
Now it works again no problem. The problem was never bandwidth it seems, for me, but something about how the airplay sync is working for audio in firmware 7.6.4.