Where does iBooks store the highlights and notes?

Question

Level 1

8 points

Where does iBooks store the highlights and notes?

I've spent a lot of time making highlights and notes in a book in the iBooks application and I'd like to get my annotations off the iPad and onto the Mac.

I know I can read them on the iPad but now I want to edit, filter and sort them.

I know that the annotations are stored somewhere on the Mac because they are synchronised between my iPad and my iPhone. I don't expect the annotation file to be easy to read but I'm happy to slog away at it so I can get my notes into an application like Numbers or Excel so I can sort and filter the notes.

I've tried looking in the epub file itself, but they're not stored there. Any ideas?

iPad 32 GB / MacBook Pro 2.66 GHz, iOS 4

Posted on Jul 29, 2010 8:48 AM

Reply

Answer 1

Jul 29, 2010 10:03 AM in response to davidtravis

David,

I don't have an answer for you, but I'm wondering about the same thing. My long-term intention is to mainly mark up PDFs rather that ePubs --- in my case, journal articles and scanned books, for example --- but I am worried about starting such a project (that is, pepping those books for transfer to an iPad, which is a lot more work that you might imagine) if --- presuming Apple ever permits marking up PDFs, which everyone in academia, at least, is clamoring for --- one can't get their text-based mark-up off the dang device along with some reference to the document and page(s) those annotations are associated with. And if one can't do it with ePubs, as is the case currently, then surely it will be impossible for such a facility to work with PDFs. I sure hope that Apple is paying attention to the education/research market and will, through the feature being discussed here as well as a handful of others, turn the iPad into an actual productivity tool rather than keep it at the near eToy status that it now enjoys.

Reply

Answer 2

Aug 27, 2010 1:38 PM in response to davidtravis

Well, there is something you can do, but it's a bit of a pain. This is what I tried:

1. Set up iTunes to sync books and make sure that "Notes" are also sync'd (check the box under the Info/Other tab after selecting your iPad).
2. get a shareware program to extract files from these "backups" - for example can get from http://supercrazyawesome.com/ -- use this program and select the "iBooks" app name to extract and put it in it's own folder
3. Get a SQL browser program to look at the "UserData...sqlite" file just extracted -- I got one at http://sourceforge.net/projects/sqlitebrowser/files/sqlitebrowser/2.0%20beta1/sq litebrowser200_b1osx.zip/download
4. Put the Browser program in "Browse Data Mode" and Select the "ZBKBOOKMARK" table and your notes should be in the "ZANNOTATION" field.

Kind of ugly, but one of these days someone will hopefully write an app that does this directly. One thing that I've noticed is that notes that have used the "Return" key to put more than 1 line in the note, don't seem to work correctly. However really long notes on the same line seem to be OK.

So, just to kind of explain a bit - iBooks does not modify the "epub" document, it merely keeps track of your "additional" info in another SQL database that tracks "where" in the document your info belongs and what it is. This can be a bit obtuse, as I don't think it can keep track of page numbers as they can change depending on font size, so not sure how it really does this, but you can see some of the pieces of info that it keeps track of, like the sentence where the "note" is located and the word within that sentence - am guessing that it keeps track of which word number location the "note" is at, as this would not change no matter what the font size is, but perhaps someone else can explain this.

Good luck and hope this helps a little ...

-Bob

Reply

Answer 3

Aug 27, 2010 4:48 PM in response to DeepYogurt

Had a bit more to add --

And also this doesn't seem bullet proof - as I keep noticing that sometimes things are missing, but can't figure out why. So there must be something in these steps that is not quite right, or something additional needs to happen to make sure things are up-to-date.

I think I just figured out how to get the correct information - I had a lot of old backups in the ~/Library/Application Support/MobileSync/Backup/xxx folder (xxx is the ID for the iPad - as there is a Backup/xxx folder for each Mobile device you have or had), so I moved all but the most recent ones to a separate folder for the moment and then did the extraction, and got the expected and most recent Notes I had created. Not sure what else this might have an affect on, so if this is also a problem for you, you might want to move them back after doing the extract so the iPad backup process doesn't get too confused.

Also, just a note about the "ZANNOTATION" field - if it's blank, I think this denotes a "Highlight" in an iBook or a true Bookmark - wonder how to tell the difference? Also, now that I've cleaned up the Backup/xxx folder, there is no longer any problem with multi-line Notes. They show up just fine. I've asked the developer of the SuperCrazyAwesome extractor program about this and will post a reply when I find out.

-Bob

Reply

Answer 4

Feb 3, 2012 11:09 AM in response to davidtravis

Just Found a better way to Export Highlights and Notes

if you have a mac.

Get iExplorer here

http://www.macroplant.com/iexplorer/

plug in your iOS device and use iExplorer to

look in the books file and drag the following files to your desktop.

iBooks.plist and iBooksData2.plist (all highlights are stored in the iBooksData2.plist file)

and this app

https://github.com/lgoss007/BookmarkExtractor

put the files in the appropriate spot and you can get your highlights

sorted by book

Reply

Answer 5

Sep 3, 2012 5:38 AM in response to Nathan005

Nathan005 wrote:

Just Found a better way to Export Highlights and Notes

if you have a mac.

Get iExplorer here

http://www.macroplant.com/iexplorer/

plug in your iOS device and use iExplorer to

look in the books file and drag the following files to your desktop.

iBooks.plist and iBooksData2.plist (all highlights are stored in the iBooksData2.plist file)

and this app

https://github.com/lgoss007/BookmarkExtractor

put the files in the appropriate spot and you can get your highlights

sorted by book

How do use this "bookmarkextractor" tool? I don't see any dmg files, just raw code files. Is that what you're suggesting, that we download raw code and execute it like programming geeks every time?

Reply

Answer 6

Sep 4, 2012 6:31 AM in response to Cyberpundit

nope you don't have to geek it up, its open source so others can edit if they want to help.

the app is under the downloads section here. no dmg, just the app

https://github.com/lgoss007/BookmarkExtractor/downloads

Reply

Answer 7

Sep 4, 2012 8:09 AM in response to Nathan005

Thanks yes it works. It's a bit too many steps to get to the highlights instead of just pressing a button in the book to "Email highlights" (as it's possible with Notes), but well this is better than nothing.

Thank you.

Reply

Answer 8

Sep 4, 2012 2:34 PM in response to Cyberpundit

yep wish it could be less steps but that would have to be done by apple.

my guess is the publishers don't want it to happen.

Reply

Answer 9

Annalea

Level 1

5 points

Nov 14, 2012 9:34 AM in response to Nathan005

I can run the app (I've got Mountain Lion on an intel iMac), but when I choose my files and click "run", the file paths disappear, and nothing appears where one would suppose the notes, et al, should appear. Any ideas?

Reply

Answer 10

lgoss007

Level 1

0 points

Nov 15, 2012 10:28 AM in response to Annalea

Two ideas.

One you could try extracting a file called "purchases.plist" and use that instead of "Books.plist", which I've been told couldn't be extracted with iExplorer (maybe the newest version will do it?), but someone told me that DiskAid could do it. Apple seems to have a bunch of places littered with book data.

Two you could send me your Books.plist and iBookData2.plist and I could see if I can figure out what the problem is (I wrote the BookmarkExtractor app for my brother). My email is my username, at gmail.

Also it might be good to write Apple and tell them it would be a good feature in iBooks to show only highlights, annotations, etc. That's where the feature really belongs, everything else is just a work around.

http://www.apple.com/feedback/ibooks_ios.html

Reply

Answer 11

Nov 24, 2012 9:16 PM in response to lgoss007

I just discovered, that this is now possible in the latest iBooks (I tested with iOS 6)

In the Notes section, tap the Share Icon, choose Edit, Select All, then email it.

Work amazing, even keeps the highlight color and context in tact.

Thanks for your help and work bro!

Reply

Answer 12

Nov 28, 2012 4:00 PM in response to Nathan005

I confirm. The message you receive is fantastic: the whole paragraph containg the highlighted text, with the title of the chapter and any eventual note. As usual, Apple has done it in the most elegant way.

Paolo

Reply

Answer 13

Apr 6, 2013 11:01 AM in response to DeepYogurt

Just to add some insight into how notes work: the highlights and notes must indeed be based off of the word count. While working with proofs (I publish iBooks), I noticed if I added pages in between the book, annotations later in the book were shifted backwards to what I guessed must have been the same number word, thus misplacing it. I would really love to see an updated ePub format that can store annotations.

Reply

Answer 14

Jun 9, 2013 9:52 AM in response to davidtravis

Just to update this thread: I've just tried Digested (an app by Crispy Driven Pixels) to extract notes and highlights from iBooks. It's ridiculously easy to use:

1. Download the Digested.dmg file and install the app

2. Connect your iPad (or iPhone) to your computer via USB

3. Launch Digested.

Digested instantly collect all highlights notes from your iBooks library, and presents you with two export options: PDF or to Evernote format. One click produces the export file.

The Digested app is availble directly from the developers' website, and is supported: http://drivenpixels.com/digested

And, it's free!

My only problem now is deciding how to manage the notes. I'm not using Evernote yet, so I exported to PDF, which means I have a rather large PDF document with highlights and notes from at least a dozen books. I could manually break this down to individual documents, but I suspect the Evernote option may be easier. So, this still isn't a perfect solution, but it is effective, fast, and easy for extracting notes.

Ultimately, I want a book-by-book solution, but this will do for now.

Reply

Answer 15

Jun 9, 2013 10:34 AM in response to Bentobabe

You don't need any app anymore. iBooks has this feature inbuilt. Try it, it's hidden in Notes.

Reply