I've had similar ghost since upgrading to iOS 7.1x.
Makes any Gem game nearly impossible to play.
Reading here, others are seeing like issue, while mine is 2nd Apple Store replacement Mini.
I do a top button cycle to clear input buffer. Eventually, it settles down just enough to continue.
I am not so sure it is only Hardware failing, but willing to concede if proven otherwise after calling AppleCare support.