How to recover a corrupt and locked Time Machine .sparsebundle

I have not been able to find a complete HOWTO for recovering a corrupt TM image. So here is the procedure I used. Hopefully it will be helpful to others.

One of my machines ate it's TM image this weekend.

1/1/11 10:35:58 PM com.apple.backupd[47630] Starting post-backup thinning
1/1/11 10:36:04 PM kernel hfs: cat_updatelink: couldn't resolve cnid 8933822
1/1/11 10:36:04 PM kernel hfs: cat_updatelink: couldn't resolve cnid 8933823
...
1/1/11 10:36:24 PM com.apple.backupd[47630] Deleted backup /Volumes/Backup of tengoku/Backups.backupdb/tengoku/2010-12-31-215532: 415.10 GB now available
1/1/11 10:36:24 PM com.apple.backupd[47630] Post-back up thinning complete: 1 expired backups removed
1/1/11 10:36:24 PM com.apple.backupd[47630] Backup completed successfully.

It would be nice if TM actually warned you there is a problem here, but alas it doesn't!!

Next mount:

1/1/11 11:36:13 PM kernel hfs: cat_delete() failed to delete thread record on volume Backup of tengoku
1/1/11 11:36:13 PM kernel hfs: Runtime corruption detected on Backup of tengoku, fsck will be forced on next mount.
1/1/11 11:36:13 PM kernel hfs: cat_updatelink: couldn't resolve cnid 8931951
1/1/11 11:36:13 PM kernel hfs: cat_delete() failed to delete thread record on volume Backup of tengoku
1/1/11 11:36:21 PM com.apple.backupd[47722] Deleted backup /Volumes/Backup of tengoku/Backups.backupdb/tengoku/2010-12-31-225531: 415.10 GB now available
1/1/11 11:36:21 PM com.apple.backupd[47722] Post-back up thinning complete: 1 expired backups removed
1/1/11 11:36:21 PM com.apple.backupd[47722] Backup completed successfully.

Could have also warned me here!

Next mount:

1/2/11 12:35:19 AM kernel AFP_VFS afpfs_mount: /Volumes/Time Machine, pid 47844
1/2/11 12:35:21 AM com.apple.backupd[47843] QUICKCHECK ONLY; FILESYSTEM DIRTY
1/2/11 12:35:22 AM com.apple.backupd[47843] Runtime corruption detected on /Volumes/Time Machine/tengoku_0017f20cbe46.sparsebundle (fsck_hfs -q termination status: 3)
1/2/11 12:35:22 AM com.apple.backupd[47843] Network destination already mounted at: /Volumes/Time Machine
1/2/11 12:35:22 AM com.apple.backupd[47843] Backup verification failed for image /Volumes/Time Machine/tengoku_0017f20cbe46.sparsebundle!
1/2/11 12:35:22 AM com.apple.backupd[47843] Moved previous backup image to /Volumes/Time Machine/tengoku 0017f20cbe462011-01-02-003522.sparsebundle

At this point what I had is a locked sparsebundle. Even after removing the lock (via Finder, Get Info) the image would only ever mount read-only.

mini:$ hdiutil attach -nomount -noverify -noautofsck -ignorebadchecksums tengoku 0017f20cbe46munched.sparsebundle/
/dev/disk2 Apple partitionscheme
/dev/disk2s1 Apple partitionmap
/dev/disk2s2 Apple_HFSX

mini:$ diskutil info /dev/disk2s2
Device Identifier: disk2s2
Device Node: /dev/disk2s2
Part Of Whole: disk2
Device / Media Name: disk image

Volume Name: Backup of tengoku
Mount Point:
File System: Case-sensitive Journaled HFS+
Owners: Disabled

Partition Type: Apple_HFSX
Bootable: Not bootable, can be made bootable if reformatted
Media Type: Generic
Protocol: Disk Image
SMART Status: Not Supported
Volume UUID: C5028CF1-9370-32D7-B5FE-F01E75930B0F

Total Size: 931.3 Gi (999957381120 B) (1953041760 512-byte blocks)
Free Space: 0.0 B (0 B) (0 512-byte blocks)

Read Only: Yes <<<<< Problem here.
Ejectable: Yes
Whole: No
Internal: No

If anyone knows how to clear the r/o flag, please let me know. Otherwise, here are the steps to fix things up.

---

1) Before you let Time Machine create a new image, save copies of:
- /Library/Preferences/com.apple.TimeMachine.plist
- The corrupt .sparsebundle

"Duplicate" via the Finder is the easiest way to make a copy of the .sparsebundle.

2) Use hdiutil to make a copy of the corrupt TM image contents to a new image.
- hdiutil convert corrupt.sparsebundle -format UDSB -o new -verbose

"corrupt.sparsebundle" should be replaced by the actual name of your image. This will create a "new.sparsebudle" which is read/write and has the same UUID and contents as the original.

3) Fix the corruption in the new.sparsebundle image. You can use fsck_hfs, Disk Utility.app, Disk Warrior, etc. I used Disk Warrior. Of course, if the image can't be repaired or so much of it is corrupt that it is not going to be of any use, you can skip the rest of the steps. 🙂 If one tool doesn't work, you can repeat step 2 and try a different tool.

4) Let Time Machine start over with a new backup image. When it has finished, turn off Time Machine
in System Preferences.

5) Delete the new image Time Machine just created.

6) Put the original /Library/Preferences/com.apple.TimeMachine.plist back you saved off in step 1.

Now, if you let TM make a new image before finding this thread, you can do the following.
- Open /Library/Preferences/com.apple.TimeMachine.plist in Propert List Editor.app from XCode
- Attach the new.sparsebundle image: "hdiutil attach -nomount -noverify -noautofsck -ignorebadchecksums new.sparsebundle"
- Get the UUID: "sudo /System/Library/Filesystems/hfs.fs/hfs.util -k disk2s2"
- replace "disk2s2" here with the output of hdiutil if different.
- Change "DestinationUUID" in the .plist to the UUID hfs.util reported.
- Detach the image.

7) Copy com.apple.TimeMachine.MachineID.plist from the original corrupt image to new.sparsebundle
- cp corrupt.sparsebundle/com.apple.TimeMachine.MachineID.plist new.sparsebundle/

8) Edit new.sparsebundle/com.apple.TimeMachine.MachineID.plist and change "VerificationState" to "1".

9) Mount new.sparsebudle at least one time and let fsck_hfs finish. This may take a while.
Then unmount the image.

10) Move the original corrupt .sparsebundle and any other copies you have into a sub directory so Time Machine doesn't find them.

11) Rename new.sparsebundle to the same name as your original image.

12) Then "Enter Time Machine". It should mount clean and have all you backups.

13) Then "Back Up Now". It should be able to complete a backup.

14) Turn Time Machine back on in System Preferences.