Alsoft's Disk Warrior still can't rebuild APFS disks.

Emailed sales at Alsoft yesterday, got this reply:

"Thank you for contacting Alsoft. We are working on an update, but please understand that the APFS file system is the first file system change in 20 years. Because HFS+ and APFS are so different, it will take us some time to update the DiskWarrior application to be compatible with APFS. Just last week Apple provided the documentation that we need to safely repair an APFS disc. Yes, we (and all other developers as well) have been waiting on this documentation so that we could safely repair an APFS disc. If you visit our web site, you will see that we have updated it with information for Mojave and also the pending release of DiskWarrior 5.2.

https://www.alsoft.com/DiskWarrior/highsierraapfs.html

DiskWarrior 5.2 will more gracefully deal with APFS. The Preview application in DiskWarrior will gain the ability to copy files to APFS disks when recovering data from a failing Mac OS Extended (HFS Plus) disk. The next major release of DiskWarrior will include the ability to rebuild APFS disks — this will be DiskWarrior 6"

VikingOSX, all three major developers of Mac utilities to rebuild disk directory indicate that the lack of such essential feature for APFS in their utilities is becasue Apple has not yet released APFS specifications:

- Alsoft - DiskWarrior.

- Micromat - TechTool Pro.

- Prosoft Engineering - Drive Genius.

Micromat has done some progress on such direction, because they have access to part of such APFS specifications, since they have a long history of business relationship with Apple to develop TechTool Deluxe (for AppleCare customers only).

The current situation of lack of information about APFS from Apple is described here:

APFS technical documentation

https://apple.stackexchange.com/questions/315367/apfs-technical-documentation

So much, that APFS has been reverse-engineered and published in prestigious scientific journal:

------

Digital Investigation - Volume 22, September 2017, Pages 107-132

Decoding the APFS file system

Kurt H.HansenFergusToolan

Abstract

File systems have always played a vital role in digital forensics and during the past 30–40 years many of these have been developed to suit different needs. Some file systems are more tightly connected to a specific Operating System (OS). For instance HFS and HFS+ have been the file systems of choice in Apple devices for over 30 years.

Much has happened in the evolution of storage technologies, the capacity and speed of devices has increased and Solid State Drives (SSD) are replacing traditional drives. All of these present challenges for file systems. APFS is a file system developed from first principles and will, in 2017, become the new file system for Apple devices.

To date there is no available technical information about APFS and this is the motivation for this article.

https://doi.org/10.1016/j.diin.2017.07.003

https://www.sciencedirect.com/science/article/pii/S1742287617301408

------

As they state, "To date there is no available technical information about APFS and this is the motivation for this article". So, here we are. Hopefully, Apple will release such information soon. Otherwise, we will not update from Sierra to newer macOS versions, since that is a deal breaker issue. DiskWarrior has saved our disks many times, including prevention and Time Machine catastrophic corruptions. A real jewel.

We have the same situation here. What Alsoft states is:

"The next major release of DiskWarrior will include the ability to rebuild APFS disks. Our developers are waiting for Apple to release the final APFS format documentation in order to safely rebuild APFS disks".

That was almost a year ago (since 28th September 2017). I wonder why Apple does not release full APFS specifications, even if that is a work in progress. That would be better than nothing.

Additionally, we are also waiting for Apple Time Machine 2 (or whatever called) with full APFS support, including snapshots. What is Apple waiting for?

I do not mind waiting, but I do mind the lack of information from Apple.

I wrote to Alsoft in late February this year (2018). It's now September 8, 2018 and still nothing, which I consider inexcusable. Here's the dialog I had with Alsoft...

Me to Alsoft (1st email):

I've been thinking about placing an Upgrade order for DiskWarrior 5.0 (since I own a previous version), but I see it comes on a Flash drive and I also see that version 5.1 is forthcoming. Since I am running High Sierra on all my Macs (which have SSDs and therefore APFS), version 5.1 would be best for me. Do you have a geneal idea of when version 5.1 will be released?

Alsoft to Me (1st reply, from Marc Moorash, Alsoft Technical Support):

For all publicly available information regarding DiskWarrior 5.0 and macOS 10.13 (High Sierra), as well as the new Apple File System (APFS), please visit: (Alsoft website & slash highsierraapfs.html)

Me to Alsoft (2nd email):

Your generic reply to me indicates you did not read the email I sent to you. You merely sent me a link to a web page on your site regard "version 5.0" that I already had read. Therefore, I shall ask my specific question once again in hopes of a more specific answer from you this time...

DO YOU HAVE A GENERAL IDEA OF WHEN DISKWARRIOR "5.1" WILL DEBUT?

My apologies for the all caps, but I want to ensure you can spot my question this time.

By the way, I am fully aware that the web page you linked for me says this:

"What can you expect soon"

But that web page has been up for a long while, which isn't the definition of the word "soon," hence my more specific question.

If you honestly have no idea when version 5.1 will debut and expect it to take a half-year or more (or even never), please simply say that. I value complete honestly and frankness.

Alsoft to Me (2nd reply, this time from Amy Smith, Customer Service):

Thank you for your email. We wish that we could provide more information, but we do not have a timeframe on when the software will be released. We are working on an update, but please understand that the APFS file system is the first file system change in 20 years. We still have not received all the documentation that we require from Apple. Believe me when I say that we want the update as much as you.

If you were to purchase DiskWarrior 5.0, the upgrade to version 5.1 will be a free updater. However, the DiskWarrior 5.1 updater will only recognize the APFS drive. Our next major release will include the ability to repair an APFS drive. When a major release is made available, you would be responsible for the upgrade cost.

Once the update is available, we will have details on our web site.

Me to Alsoft (3rd email):

Thank you for providing details, Amy!

One clarification is needed though. What do you mean when you say "the DiskWarrior 5.1 updater will only recognize the APFS drive"? Are you saying DW5.1 will only fix APFS drives and not be able to fix HFS+ drives? If not, what do you mean?

Alsoft to Me (3rd reply, from Amy Smith, Customer Service):

Thank you for your reply and sorry for the confusion. Currently DiskWarrior 5.0 is unable to recognize the APFS drives. With DiskWarrior 5.1, we will be able to recognize them, but still be unable to repair them. It will be our next major release — which is still future away that will be able to repair APFS drives.

Me to Alsoft (4th email):

Thank you for the explanation. Please allow me to explain the source of my confusion. What good is it to merely "recognize" APFS volumes if DiskWarrior 5.1 then cannot do anything at all with them? It other words, even if your upcoming version 5.1 can see an AFPS volume whereas version 5.0 cannot, the end result is still exactly the same -- nothing at all can be done to APFS volumes. And in either version, HFS+ volumes could be rebuilt as usual. Isn't that correct?

Alsoft to Me (4th reply, from Amy Smith, Customer Service):

Thank you for your reply. Please remember that I am just in sales and I do not have the technical background that our programmers have. Currently DiskWarrior 5.0 is not able to even recognize an APFS drive. The minor update of DiskWarrior 5.1 will allow Diskwarrior to recognize and also provide other needed minor changes. When the update is released, we will have more information about the changes that it includes.

Me to Alsoft (5th and final email):

I see. Well, I appreciate your time, Amy.

And there you have it.

Maybe I should email them again after Mojave is released? Then again, they'd probably reply, "Sorry but MacOS X Mojave is too new and our developers need time to assess the changes."

I really love DiskWarrior, and nothing, not even TechTool Pro, beats it at the directory rebuilding game. But this waiting game is a game none of us wish to play. Apple should buy DiskWarrior and build it into Disk Utility!

I recently read on Ric Ford's Macintouch about the excellent APFS overview by Tim Standing of SoftRAID, which mentions his contact with Alsoft (DiskWarrior), Micromat (TechTool) and other drive utility makers and the challenges they've faced due to lack of official APFS information:

http://docs.macsysadmin.se/2017/video/Day3Session4.mp4

The presentation is a wealth of great information that, while a tad technical, is good enough to warrant watching the entire video. This video was from the 2017 MacSysAdmin sessions. The great news is that MacSysAdmin 2018 is scheduled to take place next week, which no doubt will shed further light on APFS and the state of drive utilities.

Also note in that video how much faster HFS+ is than APFS, even when used on an SSD, due to the file fragmentation inherent to AFPS. So it would appear that file defragmentation utilities that we all used many years ago may return in popularity due to the nature of APFS.

If you go back ten years, Alsoft was blaming Apple for the delay in releasing the boot specifications for a version of OS X that they needed to boot from the Alsoft DVD on newer Macs. Last year, it was Hurricane Harvey that impacted Alsoft's (they are just north of Houston) delivery schedule of APFS support. Now it is a purported delay in Apple's release of the APFS specifications.

Just because a vendor runs home to “it's Apple's fault,” does not make it entirely the truth, according to the gospel of 1's and 0's.

I think Alsoft is suffering from aiming paralysis. Apple will always be tweaking refinements into APFS, but the filesystem has been available to developers for two or more years now.

I posted a comprehensive answer but it no longer shows. Here is a summary:

1. You have the answer to your question in the MacObserver link above, just clicking the person's name link.

2. In relation to this thread about "Alsoft's Disk Warrior still can't rebuild APFS disks", there is interesting information searching Internet with Google:

- Search for: Mac Sys Admin 2017 "Dive into APFS" PDF

and see pages 43, 44 and 45 of Day3Session4.pdf

- Search for Rusty Little (Alsoft/DiskWarrior)

in

disk utilities

https://www.macintouch.com/community/index.php?threads/disk-utilities.697

Also found searching for DiskWarrior-Proj-Mgr

in

DiskWarrior

https://www.macupdate.com/app/mac/11707/diskwarrior/reviews

That should clarify the situation to everybody interested on the main question of first post of this thread.

All I get back is a reference to a web page https://www.alsoft.com/DiskWarrior/highsierraapfs.html

which says that their latest version can recognize but not repair APFS disks.

Well you've heard it straight from the horse's mouth. They know more than we do so we can't answer why they don't have a compatible version. I guess we'll just have to wait and see.

I used to run DW in the "old" days; at that point, it wouldn't even reliably be able to boot back into the internal drive - the only way to use it was to have it on an external bootable clone and run it from there.

As the Mac OS has gotten updated and refined, I have found less and less occasions to actually use it and had actually forgotten that I still have it installed. Haven't had the need to run it in at least 2 years. I'd also question the necessity of using it on an SSD which, if it has decent garbage collection, should not be necessary (in my humble opinion).

In other words, I see no reason to run it.

Rebuild directory using the current version of DiskWarrior 5.1 and you will see what we mean. Unless your disks do not have many items and you do not save/delete many items in them, chances are that you will be shocked to see what you find.

As said before, DiskWarrior has saved many disks from losing all their data, as well as preventing them to became corrupt, which in case of Time Machine has catastrophic consequences of losing all previous backups and starting backup from scratch.

Again, that happens at least and for sure if you have many items and change them often on large-capacity disks, including mechanical-rotational ones and SSD. That is our experience with thousands of them for decades, up to macOS 10.12.6 (16G1510) Sierra on HFS+ and before. We have not tested newer macOS and APFS so far as described above because the lack of DiskWarrior feature to rebuild APFS. Hopefully it will be soon available when Apple releases its specifications.

Sometimes the symptoms are obvious; for instance, disk does not mount and all content is lost. I have had some cases of that from people that did not care about maintaining their Macs. Sometimes it is too late then; sometimes DiskWarrior can fix it then. I have saved some disks that way. It depends on the degree of damage.

But that is the most extreme situation. Usually, you may not notice the symptoms at all. But if then the disk directory is corrupt, you could overwrite a file and lose it for ever. And you will never notice until you need such file. How could that happen? It may not be frequent, but it can happen when you must force shut down the Mac because got frozen, kernel panic, etc, even turned it off or unplugged from mains without noticing it was on (yes, people do that; amazing but true). It may happen when you boot from external disk and the display goes black, yet it is not fully black (you can only notice that if there is not much light), and then unplug the disk to use it in other Mac (booting at work and home from the same external disk like Samsung Portable SSD T5, for instance) etc.

Or maybe it was a neutrino (no joke intended) that hit the Mac... I do no know... but disk directory corruption is a fact. For any disk, and mostly for Time Machine ones and booting ones with millions of files and big sizes... These things are not science-fiction in my troubleshooting experience of many years with many Macs. It is curious to read people saying that Time Machine disks get corrupt after six months or so... Probably they never rebuilt their directory. Maybe that is less likely or not happen at all with APFS, but I must see it. I guess it will probably happen less frequently, but still will happen, at least under extreme situations like forced Mac shut down.

And additionally, DiskWarrior also optimizes disk directory index to improve speed, checks files and folders for damage and potential compatibility problems, as well as testing hardware. But freedom is great, so the decision is yours. I bet you will never use it.

Yes, here we are at Mojave release and still no way to rebuild a High Sierra disk (with APFS anyway, it would seem to me a radical idea to install High Sierra but then convert my disk from APFS back to HFS+ format).

I'm still on Sierra as I want to have an ability to rebuild my directory if needed. DiskWarrior saved me once before in a situation where I had my Macbook Pro wasn't able to boot (SSD) and I really don't wish to be without it.

(Also, though a peripheral issue, I use Diskwarrior every few months on my Time Machine when it slows drastically down... when I do that the directory is showing 5 or 6 Directory Optimization index, which surprises me that it can be so low.. why would the Time Machine process create that low level?)

Frustrating that Alsoft can't give us any time frame at all, e.g. months, years or decades.

Garbage collection takes place if you have TRIM enabled; I do not know enough about DW to say whether or not anything would be done that DW used to do. My personal opinion is that it is (or was) quite good at rebuilding a directory, but that was about it. I certainly did not use it monthly - at most maybe 2 x per year and only if/when I felt the mac was sluggish. I have not experienced anything sluggish for a very long time. But, I am also not tech savvy enough to know for sure. I've also opted out of running HS - it did a number when I tried to install it to my 2015 MBA (black screen and no response to anything) and it took me 3 days to get it to boot from one of my clones - at that point, I decided Sierra was just fine.

DW recommends rebuilding the disk catalog monthly

Why? What problems do you see appearing if you don't? I've never done any disk repair in 30+ years of using various Macs. Never noticed any problems.

Are you saying that an SSD with High Sierra on it could be successfully repaired by Apple's own Disk Utility without the need of DiskWarrior?

I have managed without need of any repair in 30+ years, whether it be Disk Utility or another product. The only drive I attempted to repair turned out to be failing--known issue with that particular Seagate drive.

Maybe I'm missing something. What symptoms should I be looking for that would indicate I need to perform any disk maintenance? I've never noticed anything obvious.