APFS woes on external drive - failed disk / drive repair

Question

Level 1

11 points

APFS woes on external drive - failed disk / drive repair

Backstory

I've been successfully using an external Samsung 1tb - EVO SSD via USB3 with macbook pro for over 6 months. I boot to this device. Last week the usb cord came disconnected - (at the time, I was troubleshooting other macs and wasn't paying close attention) I fatally plugged it back into usb port only on other side of mac causing drive to fail. This began a week long deep dive into apfs drive corruption.

Symptoms

Started file system verification on disk3

Verifying storage system

Performing fsck_apfs -n -x /dev/disk2s2

Checking volume

Checking the container superblock

Checking the EFI jumpstart record

Checking the space manager

error: (oid 0x8790) cib: invalid o_xid (0x63f69)

error: failed to read spaceman cib 0x8790

Space manager is invalid

The volume /dev/disk2s2 could not be verified completely

Storage system check exit code is 0

Finished file system verification on disk3

**+-- Container ERROR -69808

======================

APFS Container Reference: disk4

Size (Capacity Ceiling): ERROR -69620

Capacity In Use By Volumes: ERROR -69524

Capacity Not Allocated: ERROR -69524

|

+-< Physical Store disk3s2 B8843099-2B4F-4D1B-909A-DB5B1B516B9C

| -----------------------------------------------------------

| APFS Physical Store Disk: disk3s2

| Size: 999666946048 B (999.7 GB)

My findings that came out from this is -

APFS seems to be only suitable for internal Apple drives only which provide ECC protection.

Data corruption can't be successfully recovered as Apple hasn't disclosed internals of apfs mechanics to other vendors / eg. diskwarrior. https://www.alsoft.com/DiskWarrior/index.html

APFS AFRO.

I found this tool which can actually read partitions to recovery directory / files

https://github.com/cugu/afro

While I haven't had success with above tool - I did have success using disk drill to dig out buckets of files by file type but lost file/folder structure in process. Now I'm at a dead end with attempts to fix partition.

TAKEAWAY

Maybe someone from Apple APFS engineering team will read my plight use case above, try this simple use case of of disconnecting and reconnecting usb to other port and come up with a fix in disk utility. Shouldn't APFS handle this case and be able to recover? Or perhaps consider an improvement to UX where OS would reboot instantly preventing data corruption. Or perhaps discourage the use of APFS on external drives which don't offer ECC protection out of the box, as my understanding is APFS can't do error correction as ZFS offers out of the box.

MacBook Pro with Retina display, iOS 11.4

Posted on Jul 27, 2018 7:24 AM

Reply

Answer 1

johndpope Author

Level 1

11 points

Aug 1, 2018 8:29 AM in response to johndpope

Dear Apple, please acquire iBoysoft and their software and add APFS lost partition recovery logic to your latest OS.

fsck_apfs is inadequate to repairing drive. Disk utility is not helping here.

I'm finally getting somewhere here after a week of dinking around. To any one with failed drive - I HIGHLY recommend cloning drive to an image before using this software. You can use DD to dump drive out.

Reply

Answer 2

johndpope Author

Level 1

11 points

Jul 30, 2018 8:22 AM in response to etresoft

Hi Etresoft, I agree with most of your comments above.

N.B. There's no way to run latest Majove with external SSD drive and other file system due to forced migration to APFS.

One thing that leaves me with some hope that drive is not lost - is that in white paper of Decoding APFS - it states

The APFS structure potentially provides the forensic investigator

with the possibility to recover earlier container states. After several

tests, both manually and programmatically, we have been able to

recover the container from previous checkpoints and by comparing

recovered stages, we are able to discover previous exist

I have the image of apfs drive / partition visible using 010 Hex editor alongside the actual data structures reflecting said check points. It seems like if this data was fudged with earlier checkpoint values - perhaps the drive would come good - and time warp back to an earlier container state as researchers have found. Going to do some prototyping with smaller images to see if a diff between two drives illustrating the checkpoint differences can be obtained.

Reply

Answer 3

Barney-15E

Level 10

122,294 points

Jul 27, 2018 3:50 PM in response to johndpope

Was the Mac booted to the drive when it came unplugged? That would likely corrupt the drive pretty well. Plugging it back into another port wouldn't likely cause any issue. If the Mac was still running and you plugged the drive back in, I could see corruption happening regardless of the port chosen.

Reply

Answer 4

johndpope Author

Level 1

11 points

Jul 28, 2018 6:50 PM in response to Barney-15E

mac was booted, yes. I've been researching on file system internals.

I never needed to know what an inode - found this quite good explanation

LINUX Understanding inodes - YouTube

supposedly it's possible to use iBored /hex editor (undocumented / poorly explained) to manipulate sectors on disk. Going to give it a whirl. I have a cloned backup of failed drive. I've also dug up some (undocumented) apfs software from github which can traverse apfs drive.https://github.com/tienex/apfs/issues/3 But it's an up hill battle.

block 1346671 is not an object map block

naively speaking - my thinking is - can't I just write over this block?

Reply

Answer 5

leroydouglas

Level 10

199,404 points

Jul 27, 2018 11:10 AM in response to johndpope

Maybe someone from Apple APFS engineering team will read my plight use case above, try this simple use case of of disconnecting and reconnecting usb to other port and come up with a fix in disk utility. Shouldn't APFS handle this case and be able to recover? Or perhaps consider an improvement to UX where OS would reboot instantly preventing data corruption. Or perhaps discourage the use of APFS on external drives which don't offer ECC protection out of the box, as my understanding is APFS can't do error correction as ZFS offers out of the box.

Hmmm. You where running your macOS off this external drive. I suspect you had a back up strategy in place.

Not much info from Apple kb's articles

How to choose between APFS and Mac OS Extended when formatting a disk for Mac - Apple Support

Official channels to make your voice heard:

Apple Feedback http://www.apple.com/feedback

With your same Apple ID you can sign up for a free Developers Account and start a conversation with Apple engineers

Bug Reporter https://bugreport.apple.com/

Reply

Answer 6

etresoft

Level 9

56,680 points

Jul 30, 2018 7:32 AM in response to johndpope

johndpope wrote:

APFS seems to be only suitable for internal Apple drives only which provide ECC protection.

Correct.

Data corruption can't be successfully recovered

Correct.

Now I'm at a dead end with attempts to fix partition.

Don't waste your time. Erase the drive and forget about it.

Maybe someone from Apple APFS engineering team will read my plight use case above, try this simple use case of of disconnecting and reconnecting usb to other port and come up with a fix in disk utility.

Not a chance.

First of all, you were booted from an external device and then disconnected it. That's a full stop. There is no recovery from that.

Second of all, Apple doesn't care about people playing around with APFS on external drives.

Shouldn't APFS handle this case and be able to recover? Or perhaps consider an improvement to UX where OS would reboot instantly preventing data corruption. Or perhaps discourage the use of APFS on external drives which don't offer ECC protection out of the box, as my understanding is APFS can't do error correction as ZFS offers out of the box.

Why are you comparing APFS with ZFS? It seems that many people have fallen into the same trap. APFS is a new filesystem. So people seem driven to switch to APFS thinking it has some kinds of enhanced features. They know about new features in other filesystems like ZFS and assume APFS has similar features. It doesn't.

ZFS was designed for servers. APFS was designed for a watch. That's all the comparison you need.

Reply

APFS woes on external drive - failed disk / drive repair

Similar questions