I agree with others that this is an old problem and that in the last few years mac os seems to be going downhill, with many annoying bugs popping up where it shouldn't (given its price tag). I also dislike the fact that xcode is now 4.5GB which add to this problem.
However for this particular problem, i have found a way out, not sure if this is specific to my system (macbook pro mid2012, macos sierra). I notice that while the download randomly got stuck in one state or another, in this case the "waiting" state, when i look at the "network" pane in the "activity monitor", i found that the process "storedownloadd" continue to receive new network packets, which means appstore is still doing its job, but didn't know about it đ. What remains then is a way to refresh appstore. Restarting appstore wasn't enough, eventually what works for me is to put my mac to sleep for a few seconds, upon wakeup, appstore complained that the download failed, but seconds later it picks up where it had left off normally (with download status bar etc, no more "waiting"), voila đ . I guess you could also try just cutting your internet connection (to create the download failed scenario).