iBooks read-aloud freezes

I have made a couple of fixed layout books with read aloud function, which are not yet published to the iBookstore. One with word-by-word markup, and one where I have just a single SMIL par-tag per page. They seem to have the same problem. At first they work fine in iBooks, but eventually next time I test them, the read-aloud freezes at the first word or page which has a SMIL-doc attached to it. That is, it will flip through the first pages but will not get through the actual read aloud words. The only solution I found so far is to restart the iPad. Then it will work for a while, before the same thing happens again. And that is not what I would want to happen once they are published.

Both books go through epubcheck 3.0.1 just fine. What could be the problem?