Time Machine and kernel_task transferring large amounts of data

The electriclight article is interesting, but there are problems involved with sparsebundle (basis of the new Time Machine approach) and also sparseimages.

I had huge problems verifying an encrypted APFS Time Machine backup on an SMB share on a NAS/Server: https://discussions.apple.com/thread/252729592?answerId=255755542022#255755542022

The whole thread centers around how unrelieable the use of sparsebundle and sparseimage has become lately. Since sparsebundle is a centerpiece of modern Time Machine on Big Sur, the problem, that crops up, is:

If it is impossible to verify the content of the physical backup on your NAS/server, how can you be sure, that this sparsebundle based backup is even valid, if - for instance - your current Mac Mini M1 decides to enter the eternal bitfields in a fried state?

I’ve found no way to verify, that the physical content (as a standalone copy) is actually a valid backup, that can be rolled back to a new Mac Mini M1 in a working and valid state.

Any ideas to verify actual content of the physical sparsebundle is welcome!

Regards

Privacy tab allows you to exclude volumes or folders, but not TM mounted volumes. With the user interface, you can't select the mounted volumes as the file selection panel doesn't allow you to go to /Volumes. Even if you open a Finder window with the go to folder option and type in /Volumes, you will see the mounted TM volume but if you drag it to the Spotlight privacy panel, you get an alert stating "xxx volume is a TimeMachine Backup volume, you can't add it to the privacy list"

So, this is not an error. It's done by intent.

Regarding the full backup issue, as mentioned above, in my case the problem stopped after I switched from using a afp share to a SMB share. You can configure Synology to only share a volume with SMB and then you have to reconfigure your TM backup preference.

From what I see, and understand, the biggest part of the problem is really the spotlight indexing. The backup itself was acceptable. I monitored it by launching it manually with the command line above. But then backupd started indexing the TM volume with mds and mdworkers. And those processes are working by "reading" (i.e. downloading) files on the remote volumes and then writing indexed informations to it.

One could think it would be much more efficient to index things locally rather than remotely. But my guess is that it's, at least partly, related to apfs. Before with hfs+ there was a log of all file system changes that allowed you to clearly identify what were the modified files on your drive since last back. With apfs volumes, there's no longer this. There's a snapshot of changes at block level hence you don't directly get the list of files to backup. I think the whole TM process is more complicated as it has to rebuild a list of files and actual content to the backup volume. Until recently it couldn't just backup snapshots as apfs was not supported on TM volume because there's no hardlink in apfs as TM snapshot was using them to avoid duplicating files that did not change.

I'm not sure how it's done now:

• TM could Backup snapshots directly but it would be tricky to manage as any missing snapshot could preclude TM from rebuilding an exact image of all files at a certain point in time
• I don't really understand why TM now needs to index backups with Spotlight as, in the same time, it seems that Apple says they are not included in Spotlight results when you search something
• Using apfs format on the backup volume sounds like a non-sense especially with NAS using regular HDDs that are sequential by nature vs apfs that is expecting random access for efficient file access. This alone could explain a big slowdown while indexing the backup remotely if you have to go randomly through all changes in apfs snapshots on the NAS volume
• The previous point by itself suggest that the best way to go would be to rebuild a defragmented image of latest changed files on the backup snapshot but it still doesn't solve the hard link requirement of traditional TM snapshots
• Because of all the above, I think it would make more sense to continue using a HFS+ format for the TM disk image bundle on a NAS but I don't know how to do this

Time Machine and Spotlight (n-tuple mdworker) work together to index and backup to your RAID array on the Synology. That is a busy time for the UNIX kernel (kernel task) to handle those System and network requests, along with all the other customary Mac process requests in the operating system.

Continue to update macOS to 11.5.2, and see if there is any change, and hope for a Synology forum response that can help. Too many variables with your setup to pinpoint an issue with macOS, and your raid configuration may be a contributing factor.

I have a Synology NAS here and have never used its TBs of storage for Time Machine backup. Too much history in these communities with those with NAS configurations used for Time Machine and when things go wrong…

I've got the exact same issue. Time machine activated on a Synology DS920+. I switched from afp to SMB share because I regularly was facing issues with the backup and TM asking to do a fresh full backup of my MBP internal 1TB SSD every now and then.

No problem of corrupted backup since.

But, first surprising thing is that since I updated to Big Sur every hourly snapshot could be hundred of megabytes even if I'm just having simple activity like writing mails or word documents.

Then, I realized that actual network exchanges are multi GB per hour. Investigating this, I found that after each hourly backup from TM, Spotlight start indexing (reindexing?) things on the TM share reading and writing GBs of data over the network. With a lot of mds and mdworker CPU workload. It even preclude the MBP to go to sleep with assertions saying "waiting for backupd" and "waiting for backup indexing to finish", plus diskarbitrationd (for mounting the share), etc.

I also have multi tons of "Aug 16 16:58:06 MacBook-Pro-2018 com.apple.xpc.launchd[1] (com.apple.mdworker.shared.01000000-0100-0000-0000-000000000000[17701]): Service exited due to SIGKILL | sent by mds[95]" flooding the system.log.

1 process killed every 10-15 seconds, figure!

All in all, the whole backup task is taking ~20-30 minutes at a minimum, every hour, with enough CPU activity to have fans blowing all the time and slowdowns. Not to mention that even if I don't use the laptop, the display is off due to energy saver, but the MBP is warm/hot and not sleeping at all during that time.

It's such a pain that TM that was a must have option that I always used since it's introduction is now disabled by default because I can't bear to have my MBP overheating with fans blowing will I'm just doing basic work with it (office work, not 3D games or video editing)

Please, Apple, wake up... Give us back the basic and robust functionalities we always loved with our macs and stop letting interns updating core technologies making it unusable

No, you're not alone 🙂

I tried, but unfortunately you can't stop Spotlight from Indexing the backup, at least not with TM preferences. I didn't try with the command line tmutil.

I don't know for other 3rd party software la CCC or Synology own options, but what you can do is launching backups manually with command line in the Terminal.app:

tmutil startbackup

Or you can even create an automated schedule of it on a regular basis (at night for example) by creating a LaunchDaemon in your ~/Library/LaunchDaemons folder

FWIW. I have a Synology DS916+ and have been successfully backing up to it with Time Machine for a number of years. However, recently with macOS Big Sur, I started to notice the backups slowing done, and eventually, stopped working all together. Thinking it was specifically a Big Sur issue (especially with APFS), I updated macOS to every incremental version, but that still didn't seem to resolve this issue. What finally did was to upgrade the OS (DSM 7.0-41890) on my NAS to the latest version available and it has been "smooth sailing" since then.

To amend my previous post, I found some information about changes in TM strategy since Big Sur.

Apparently it now uses APFS file system by default. With dedicated drive directly attached, it apparently uses one full disk per mac as it's creating a dedicated TM Backup partition within the APFS container. And it seems it's able to optimize backup by copying APFS snapshots.

I'm still unclear about how things are done with network storage besides it's also using apfs filesystem but within disk image sparsebundle. I'll do some test when I have some time to see how are stored backed up files without hard links.

That said I've found some potential errors within my TM logs with file access problem from mds_shared due to sandbox rights on some files. I need to investigate if it could explain slowness and/or the bunch of killed mds_shared processes.

By the way, searching the web for information about these logs, I found this site https://eclecticlight.co/mac-problem-solving/ that has the most comprehensive information I found until now. There are very in depth reviews of TM process with a lot of details articles about internals mechanism and solutions. There are also a bunch of very useful tools the author has developed including The TimeMachineMecanic and Ulbow which is a universal log viewer that is much more practical than doing this with the log command.

They also have plenty of other articles on macOS internals and solutions on other typical issues you can face.

I strongly recommend that you have a look at it as I think it could give you some starting points about your own issue.

I hope it will help.

APFS is definitely the recommended filesystem format since Big Sur. Now, in support article Apple doesn't even mention the HFS filesystem format for TM. For classical HDD it could still have some disadvantage in terms of performance due to the sequential nature of the HDD. See https://bombich.com/blog/2019/09/12/analysis-apfs-enumeration-performance-on-rotational-hard-drives from the CCC creator blog for more information.

That said, you should also check what is the filesystem format used by you TM image sparse bundle on the NAS. I would recommend to keep both the same if they are old and you created them before big sur. Big sur can still use old TM archive created previously so it's compatible with backup from previous MacOS.

But if you create a new backup from scratch in Big Sur with APFS, it will probably not be compatible with older MacOS version. It 's probably different thing between direct attached TM HDD that as a dedicated partition format that was unknown before. And network shared TM image sparse bundle that could eventually be compatible with Catalina or Mojave, as the filesystem was already existing, but that TM probably won't recognize it automatically.

I hope it could help.