H.264 is an open standard (see: http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC). Note the matrix at the bottom that compares QT, x264 and other implementations. The standard is open and available to all. There are not supposed to be any secrets. Anyone with the requisite skills should be able to create applications that create and/or display H.264/AAC video.
Of course there will be some things that are undefined or ill defined such that one implementation differs from another in some small and insignificant way. An example of this is the placement of the "fast start" atom in an MPEG-4 container of H.264/AAC encoded content. The standard only specifies that this atom be at the front of the file. Apple puts it in one spot and ffmpeg puts it in another. They are close but not in the exact same byte range. Apple has even changed this location over time. Sometime in the last five years or so, Apple has moved the location of this atom from bytes 5-8 to bytes 37-40.
The issue here is the disconnect between members of the MPEG-4 H.264 open source community. This includes Apple and the ffmpeg community as well as others.
To reiterate a portion of my original post, Apple's iBooks Author application is rejecting perfectly functional .m4v (MPEG-4 container, H.264/AAC encoded content) files. The same files that IBA rejects play in the iBooks.app on an iPad when they are embedded in an ePub document created by the Apple Pages application. In other words, iBooks Author is looking for an inconsequential detail that has nothing to do with playback ability in iBooks.app on iPad and rejecting video that doesn't have this inconsequential attribute.
The iBooksAuthor application is quite young (v1.1) so this fault is most likely due to 1) overcautiousness at Apple and 2) reticence by Apple in sharing critically important information with the open source communities of which it is a member. If whatever iBooks Author is looking for that it doesn't find in Handbrake output is important, Apple should make it public. The support documents on video in iBooks Author that I have seen (2 of them) do not reveal anything beyond an *.m4v container and H.264/AAC encoded content.
Conspiracy theorists will try to make more of this.