Time machine missing files

Let me verify the validity of the assertion in the link your provided tonight.

The M$ Outlook Identity Management layer sets the extended attribute, not TM. TM just honors what Outlook sets it to be.

The extended attributes are (the highlighted value is what tell TM to exclude it).

mdls ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

com_apple_backup_excludeItem = "com.apple.backupd"

kMDItemContentCreationDate = 2014-02-08 18:18:24 +0000

kMDItemContentModificationDate = 2015-02-13 01:07:30 +0000

kMDItemContentType = "dyn.ah62d4rv4gk8y8vnekk"

kMDItemContentTypeTree = (

"public.data",

"public.item"

)

kMDItemDateAdded = 2014-02-08 18:18:24 +0000

kMDItemDisplayName = "Database"

kMDItemFSContentChangeDate = 2015-02-13 01:07:30 +0000

kMDItemFSCreationDate = 2014-02-08 18:18:24 +0000

kMDItemFSCreatorCode = "OPIM"

kMDItemFSFinderFlags = 0

kMDItemFSHasCustomIcon = (null)

kMDItemFSInvisible = 0

kMDItemFSIsExtensionHidden = 0

kMDItemFSIsStationery = (null)

kMDItemFSLabel = 0

kMDItemFSName = "Database"

kMDItemFSNodeCount = (null)

kMDItemFSOwnerGroupID = 20

kMDItemFSOwnerUserID = 501

kMDItemFSSize = 100347904

kMDItemFSTypeCode = "OEDB"

kMDItemKind = "Microsoft Outlook database"

kMDItemLogicalSize = 100347904

kMDItemPhysicalSize = 100347904

Regarding

Apple Time Machine should at least initially open with some warning for the user explaining what files, especially those from major developers (Adobe, Microsoft, etc.) might not be backed up by Time Machine.

There is no mechanism to do this. If you install TM today and start backing up, and then install Outlook tomorrow, how will TM know if this was intentional or not. There could be hundreds of such files installed by applications which cannot be listed in the System Preferences -> Time Machine -> Options list. If such a warning has to come, it should come from M$.

This does not help you at all with your data loss, but may help those who follow. To prevent such data loss in the future, there is a nice little article at which may help others.

http://hints.macworld.com/article.php?story=20080328025026826

Please also see http://forums.macrumors.com/showthread.php?t=659839

It can be removed using the following Terminal command

xattr -dv com.apple.metadata:com_apple_backup_excludeItem ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

which results in

$ xattr -dv com.apple.metadata:com_apple_backup_excludeItem ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

$ mdls ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

kMDItemContentCreationDate = 2014-02-08 18:18:24 +0000

kMDItemContentModificationDate = 2015-02-13 01:07:30 +0000

kMDItemContentType = "dyn.ah62d4rv4gk8y8vnekk"

kMDItemContentTypeTree = (

"public.data",

"public.item"

)

kMDItemDateAdded = 2014-02-08 18:18:24 +0000

kMDItemDisplayName = "Database"

kMDItemFSContentChangeDate = 2015-02-13 01:07:30 +0000

kMDItemFSCreationDate = 2014-02-08 18:18:24 +0000

kMDItemFSCreatorCode = "OPIM"

kMDItemFSFinderFlags = 0

kMDItemFSHasCustomIcon = (null)

kMDItemFSInvisible = 0

kMDItemFSIsExtensionHidden = 0

kMDItemFSIsStationery = (null)

kMDItemFSLabel = 0

kMDItemFSName = "Database"

kMDItemFSNodeCount = (null)

kMDItemFSOwnerGroupID = 20

kMDItemFSOwnerUserID = 501

kMDItemFSSize = 100347904

kMDItemFSTypeCode = "OEDB"

kMDItemKind = "Microsoft Outlook database"

kMDItemLogicalSize = 100347904

kMDItemPhysicalSize = 100347904

It can be verified that the xattr command works, by checking after a "Backup Now" and looking at the latest backup or using

tmutil isexcluded ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

[Included] /Users/MyName/Documents/Microsoft User Data/Office 2011 Identities/Main Identity/Database

I am glad to read you confirmed my understanding of what I had mentioned reading elsewhere about the tag. But even more I appreciate the explanation and demo about how to go into Terminal and change that. I learned about as much about using Terminal as I had ever known before. And I read the referenced articles you mentioned.

Next I double-checked and confirmed the TM is told to exclude backing up Outlook's Database file. I changed that, then entered the Terminal command again to verify that Database file is not among those to be excluded. Next I ran Time Machine "Back Up Now" command and found the file "Database" is now backed up onto my AirPort Capsule.

Just to see what all the fuss was about with Microsoft tagging, I copied the backup Database over and opened it, it does not open as perfectly as it might since a new message window came up with some 57 mb (as I recall) attachment. But once I deleted the attachment, then deleted the empty E-Mail form, I see that it will be possible to recover other E-Mails by exporting them from Outlook. For my money it is better to have some Database backup on TM than none.

Regardless of the limited success with TM I will also continue to use Carbon Copy Cloner on another hard drive to back up all files. I seem to have no problem reloading "Database" from the cloned drive, installing that and opening it up with no problems.

Thanks again.

PS: I do wish Apple would have some small application or some function in TM that would allow users to scan their Mac to look for and reveal the 'exclude TM' tagging. The app could be run just prior to the first Time Machine Backup, or just provide some hints of what software might not have this tagging, or be there to be run on demand. I do not claim to know the best way to implement a way to warn users, but after I lost my E-Mail and could not find it on TM I did lose some sleep over this until you and others on the Web helped me to comprehend what was going on.

Has anyone figured out what has happened. My new iMac less than 5 months old, went "gray screen". Genius bar wiped out my computer and reinstalled everything to fix gray screen. Assumed since I had a recent time machine back up all was good. Fast forward a few days and find ALL my documents are MIA. First the documents folder was missing, so the tech had me reinstall using the date before gray screen happened. The document folder reappeared and I thought all was well. Nope. No contents in the document folder. All documents are gone. Not on external and not on iMac anywhere. The tech is back to helping but even his is perplexed and has lead me to believe all is missing. I'm not sure why time machine is touted as a great feature on the apple products when it appears to have major glitches which are catastrophic. Anyone have any ideas what has happened?

Thanks for running the confirmation after the changes. I also ran the Compare, but did not test the restore, since it is my work email. 😉

martin21045 wrote:

PS: I do wish Apple would have some small application or some function in TM that would allow users to scan their Mac to look for and reveal the 'exclude TM' tagging. The app could be run just prior to the first Time Machine Backup, or just provide some hints of what software might not have this tagging, or be there to be run on demand. I do not claim to know the best way to implement a way to warn users, but after I lost my E-Mail and could not find it on TM I did lose some sleep over this until you and others on the Web helped me to comprehend what was going on.

Please see

1. http://hints.macworld.com/article.php?story=20120716005852530

2. http://support.code42.com/CrashPlan/Latest/Troubleshooting/CrashPlan_and_OS_X_Me tadata

1. After the re-install on the iMac, when TM was set up again, did it ask you for inheriting the previous backups?

2. You should periodically run tmutil compare, perhaps to ensure that you are expectations are being met vis-a-vis backup files.

3. If the TM Destination is on a non-Time Capsule device, there is something you should be aware of - AirPort base stations: Time Machine hard drive compatibility - Apple Support. This states explicitly that external non-Apple NAS storage devices are not supported. The 80211.ac model will support a USB-connected device directly connected to it.

I just wanted to add that I had the same problem. I decided to do a clean install of Yosemite and noticed a bunch of iTunes music missing. I restored from backup to see if that would bring them back but I had the same issue. I then noticed a bunch of Mail messages missing from "On My Mac". I was able to go back to September 2014 in Time Machine and retrieve the files so I was lucky. I've got a lot of photos so who knows if that's all there. Seems to be some issues from Mavericks to Yosemite upgrade possibly. This is very unnerving to see this. There must be tons of people unknowingly backing up stuff thinking it's working. Hope this gets resolved. I've always had no issues in the past.

My recommendation is to use Terminal and execute tmutil compare to see what is the delta between two backups.

It's a good idea before doing a restore or clean install. Unfortunately it's not much of use after the fact and a few years of backups. No idea when the problems started.

Yes, I quite agree with your observation.

Unfortunately, the TM issues have been there since post-SL days. They have lingered through many iterations of OS X versions in one form or other. Some users use two different tools and run two backups at least once a day to separate external storage for redundancy and recoverability. There is also a suggested weekly restore and boot exercise of sensitive data using tertiary external storage. This is a very expensive and sometimes very labor intensive process.

Loss of information causes lost trust in a tool and can raise questions regarding brand loyalty. There is enough legalese to cover and indemnify the software vendor.

I first noticed this issue last November after having run Time Machine since it was first introduced, and it shook my confidence quite a bit. I have always been able to "fix" the problem temporarily by using "touch" on the affected files and rerunning the backup, at which point they would get copied. The problem persisted through an internal drive replacement and now on a new machine, which went a few months without any detectable missing copies until I found one today.

I've been thinking of ways to streamline the "touch" workaround but then I realized that if, in order to get a reliable backup, you need to run tmutil compare, modify some files and potentially run another (quick) backup EVERY TIME then what's the point? I think I'm going to rely 100% on cloning solutions with an archive of changed files from here on out, which I've been running alongside TM anyway since I first hit this issue.

I filed a bug on bugreport.apple.com when I first hit this, and it's still open. I hope others have filed bugs as well. It'd be great if this could be fixed at some point but it would obviously take some time for TM to build trust again.

After restoring a Time machine backup this morning I have hundreds of missing documents, going through the time machine backups manually even though I have tens of backups dated yesterday, the last time my scanned documents were backed up is over a week ago. I've lost 10 hours worth of duplex scanning.

Time machine is worthless and I wouldn't trust it from this point forward.

I usually run tmutil compare to get a listing of files backed up. It may not help you now, but may be worth exploring further.

Thanks, there are so many ways I could have recovered my machine and not lost the data, the thing is I believed Time machine had the data backed up. Now I know Time Machine cannot be trusted to backup my data I'll change the way I handle it altogether, boot from another drive first and lift off any data if needed but I'll keep critical stuff off the boot drive.

I just thought people should know it's unreliable, a folder I added files to daily and dated them hadn't had new files added in over 20 backups going back 8 days.

Hopefully your tip may help somebody, for anyone in my situation data recovery software didn't recover anything wiped by the time machine restore.