Issues with sparsebundle after 11.3

Are Time Machine backups in peril?

Since sparsebundle's are also TimeMachine backups, I decided to check, how safe my backups were.

How do you do that?

First I tried to extract all the content of my “Yottamaster” SSD, that is also part of my general Time Machine regime. In total 166 GB to be “rolled back” into another SSD (a bit less than 2TB free space). And this “flies”… ahem... I mean…. speed is around 10 times faster, than the 1GBit connection can deliver from the involved NAS. Not good? Not Good!

NONE - as in NONE AT ALL - of the 166GB to be “rolled back” are actually read and transferred to the SSD from the Time Machine sparsebundle on my NAS!!!

I don’t like - as in not like at all - that a backup to NAS, is restored by time machine, with roughly ten times the speed of the NAS connection. The data rolled back could exist ONLY on my local system SSD. for all I know.

The sparsebundle file may only be in the correct state, if accessed from the system creating the TimeMachine backup. If trying to roll back to a completely different machine, data may be incomplete.

Verify backups does not seem to do anything either. Here’s a screen recording showing, that probably only the local SSD is “sparsely” accessed - at best (if at all):

https://drive.google.com/file/d/1XEAI2Kr8f5DbGYstmV86f8bO-4h-rUwS/view?usp=sharing

Verify should at least display some REAL network activity exceeding single digit kByte/sec, when verifying a 1.32TB sparsebundle Time Machine backup - nothing happens. This does not bode well for the safety of your Time Machine backups, or….?

Then, to round the thing off, I decided to copy my sparsebundle from my NAS involved in this to an external SSD. Starting 08:52. This transfer seems to actually take place physically:

It is expected to last for around 4 hours, which at least seems reasonable (average ~92MByte/s should pose no problems), compared to the more than 1.1 GByte/sec shown above). So this actually reads and transfers…. ahem… At 09:02 I got this message:

After roughly 10 minutes, I no longer have the permission to read my sparsebundle. 

I don’t like, that I cannot even force a restore of the actual content from the actual backup on my server (free space on my system SSD is larger, than anything I can choose restore - even the system SSD has 603GB available for other use, so…. ehhh… even a complete “rollback” of the entire ystem to another SSD could take place from “internal snapshots” only (haven’t bothered to try, don't know if I will), that never or only partially has seen their way to the Time Machine storage on the NAS).

Anyone having any idea, script or likewise, that enables you to check the integrity of the actual encrypted sparsebundle Time Machine backup on a NAS in situ? I have found no way. I may have overlooked something, though…

For now, I regard Time Machine backups to SMB shares on servers as “unreliable”, if not downright “defective”, when it is impossible to even roll back 166 GB of content from the actual network backup.

But… the sparsebundle debacle poses some heavy doubts of the usability and validity of Time Machine Backups produced by Big Sur 11.5.1 and earlier; who knows - any and all backups produced since Big Sur 11.3 may be in doubt, if all or part of the most recent backups only exist as snapshots on your local machine.

Anyone with a suggestion on how to verify ACTUAL CONTENT of a sparsebundle file, that cannot be copied into even an external SSD with nearly 2TB free space (complete sparsebundle is 1.32TB), because the sparsebundle access rights get revoked after around 10 minutes, stopping the copy.

Currently I’m copying my sparsebundle out from the NAS to an ExFAT formatted disk connected to my Windows 10 notebook at my side. This is due after some hours, but I have my doubts as to the accessibility of the sparsebundle, when connected to my Mac Mini M1 with Big Sur 11.5.1 (I'm not updating to 11.5.2 since it still has unresolved Thunderbolt issues).

Until I've gained "alternative access" to my Time Machine sparsebundle, and the backup otherwise verified, it may pay to regard Time Machine sparsebundles (typical for backups to SMB shares on servers) as suspect. Currently, I only regard my older backupbundles (from another machine) as "maybe usable". Sigh...

Is this really the new Apple?

Regards

Addendum to: Are Time Machine backups in peril?

I have now verified, that older backupbundles (created on my Intel MacBook 13 a long time ago) actually are verified by accessing the physical contents on my SMB NAS share, as far as I can see, but.... after around HALF has been checked, there is NO real network traffic anymore. Initially the network traffic was in the region of 50-60 Mbyte/sec, but halfway up, traffic varies between a few hundred bytes - mostly - with few peaks less than 100kByte (which may be caused by other processes, like mail, calendar, contacts and Note refresh etc). Local disk (SSD) is scanned at roughly 5-25MB/sec from halfway through and onward. Sigh!

The backupbundle is 323GB, and it seems odd, that half the time of the "verify process", there is no real traffic via the network (transfers in the region of 1kByte down to 2-300 bytes per second equals "no traffic" in my eye, when 322GB data is involved ;-)

Automatic backups have been disabled, and last backup was 22:33 yesterday, more than 15 hours ago. The backupbundle was created 20. January 2021.

Regards

Kurt Friis wrote:

The backupbundle is 323GB, and it seems odd, that half the time of the "verify process", there is no real traffic via the network (transfers in the region of 1kByte down to 2-300 bytes per second equals "no traffic" in my eye, when 322GB data is involved ;-)

This is normal. In the "verifying backup..." phase, a copy of the metadata from the remote drive is made to your local hard disk. You can see the location where the copy is, using "hdiutil info". At that point, fsck_hfs starts verifying the drive, which can take a LOONNG time, even with the metadata on SSD. (For my 1.4TB backup spanning over 2 years of backups, it takes over 12 hours. During that time there is no network activity). You can follow the process, sortof, in /var/log/fsck_hfs.log, but it can still take hours for a new fsck step to show up in that logfile.

In regards to your earlier post about restoring a previous backup... note that apple also keeps local snapshots. I think you're being too paranoid, the backups are probably fine. If you want to make sure, try to restore a file or a bunch of files that you deleted a while ago (at least a week, local backups only go back days at most... use "tmutil listlocalsnapshots /" to verify.

If an around 500-600 GB snapshot (of a 1.32 TB NAS located backup) takes around ONLY one second to verify (as seen in the video), I’m not paranoid. I’m downright scared, since this can in no way involve the 1.32 TB sparsebundle present on my NAS (no network traffic shown either during the super short verify process).

At the very best, only the snapshot is involved. The physical backup on the NAS cannot be verified for any form of validity in the time frame of around one second over a 1 Gigabit network.

How can I be sure, that an old version of a file, is even present in the snapshot, and how is it verified, that the copy I may or may not need to retrieve from the NAS at some point in the future is actually valid - even at this moment. The snapshot may be valid, but is the old version physically valid on the network? Can it be retrieved, at the time of verify?

If only checksums and/or hash values stored locally on my M1 machine are compared, it says nothing about the actual content on the NAS during verify. It only says, that the historic checksum/hash value - at any historic time of generation - was correct, if equality is established.

It says nothing about the readability of the involved and encrypted 256 MB “bands” in I.e. my sparsebundle at the time of verification. That cannot be established within the scope of roughly one second.

Regards

Addendum to addendum to.... :

4 - FOUR - hours ago, the Mac Mini M1 was shut down (no AC either). It has now been started, and this time - shortly after startup, I again started the verify process. Now there us traffic on the network connection and after some minutes preparation network traffic is around 2-5 MByte/sec (network is heavily stressed by reading and creating a copy of the same sparsebundle on a Windows 10 machine).

The crux is, that this is nowhere near, what my previously produced video shows.

There may have been a bug involved, but not even an error message was presented. The positive view is, that verification may not have worked at all. Without any warning or error message. No success message either, but… if a user does this for the first time, how is (s)he to know, that one SHOULD be coming?

That’s probably, what is possible when another machine positively hieves around 100-120 Mbyte/sec out on the same network.

Let’s see, how long it takes to crunch through the task.

After around 10 minutes (plus?) - 600 seconds - all network read traffic has ceased (Time Machine backup is not active for this test). This looks suspiciously similar to the time frame involved, when a sparsebundle is copied into my machine. After 10 minutes, access is denied. I get a message, when the copy is done manually by me at the keyboard. It's time bound, not performance bound, it seems.

No error message. No success message. Up to now.

CPU System is around a couple of percent. User the same. Available: Around 96%. Not dramatic, if you acknowledge, that this note is written, while the - ahem - verification procedure is going on.

I’ll let the machine trundle along for a couple of hours, without me at the helm. If no success message is produced, sparsebundles are buggy. Time Machine also affected.

If no success message is received in a couple of hours, I’ll try a new verification operation.

Care to guess, whether that one will last one second, around 10 minutes or something else (preferably much longer or comparable to a quarter size backupbundle verification - performed on another machine - at least in time, since the same network, NAS etc. is involved, and the M1 does not speed this up in any way or form)?

Smile.

Last addendum and entry in this matter (I've decided to phase out my Apple gear due to.... guess :-)

A couple of hours later, I log in to my Mac Mini M1 via Touch ID, and now nothing happens, when I select “Verify”. Opening the menu again, gives me this result:

Stating that it is "Preparing Control….” but nothing else is shown. I can “cancel control” though (smile - Apple isn’t in control, I guess ;-)

When I open "Time Machine" some progress is shown after a looong time. There is some reading going on, very low performance (below 10 MB/sec), CPU System around 1-2.5%, User around 3-5%. Not what I would call stressed in any way or form.

There is NO activity on the Windows copy front (copy on that front has stopped around 15 minutes ago. So…. The extremely low performance is “self inflicted” by Big Sur 11.5.1 Time Machine behaviour (no restart has been performed to see, what happens, if the system is "left alone"). Short conclusion: "Time Machine is buggy".

After far, far less than 10 minutes, all activity on the network ceases (reads below 25kByte/sec to - typically - below or around 1kByte/sec).

No success message. No error message. Just no activity on my network.

I will not test any further.

I tend to get extremely miffed, if ANY company “messes” with my backups, so Apple is quickly reaching a level below “persona non grata” in my environment. The current state is completely inexcusable.

I’m looking for a different backup solution, until my Apple gear can be phased out completely. Luckily, I have a few candidates. One requires an update fee, the other is covered by an existing license.

This is my last post in this matter.

Regards

A lot of hidden file behavior is due to Spotlight rebuilding its index. Or the encryption process taking its time.

then it is entirely possible your data storage is having trouble staying healthy.

http://www.binaryfruit.com/ DriveDX

can verify your drive's health.

Have you ran (Disk Utility's) First Aid on the volume inside the disk image?

Did the spotlight exclusion help??

Have you tested it in another user account to see if the problem is system wide or limited to your account?

Addendum to Addendum to: Are Time Machine backups in peril?

How and where are snapshots stored, and in what state, where Time Machine works on encrypted sparse bundles?

Regards