Replacing Hard Drive and avoiding full Time Machine Backup

Gator TPK wrote:

. . .

I'm thoroughly impressed with your response times! I was outside and my iPhone "dinged" letting me know.

I was sitting here working on a revision of #B6 for this kind of situation. (it wasn't as complicated as I thought it would be). When convenient, please see if it's clearer now, and let me know.

I just read the "man rm" page. It's nice to have 1920 line display to read the whole "rm" manual at once, I needed to go up and down the page a bit at the different "rm" options. I read about the -r and -f options. Looks good. The -f option ignores the files permissions. So would the Sudo be needed? Or is it to override the "removal error" of "." and ".." files?

Or am I just talking nonsense and the Sudo is absolutely needed?

Also, "chflags -R nouchg " is still running. Would it be not advisable to run the "sudo rm -rf " command in another terminal window on the same package simultaneously?

Message was edited by: Gator TPK

Gator TPK wrote:

I just read the "man rm" page. It's nice to have 1920 line display to read the whole "rm" manual at once, I needed to go up and down the page a bit at the different "rn" options. I read about the -r and -f options. Looks good. The -f option ignores the files permissions. So would the Sudo be needed? Or is it to over ride the "removal error" of "." and ".." files?

I don't know -- I'm just a novice at Terminal and UNIX. 😉

Also, "sudo chflags -R nouchg " is still running. Would it be not advisable to run the "sudo rm -rf " command in another terminal window on the same package simultaneously?

Again, it sounds like it might be trouble, but wayyy out of my comfort zone!

Pondini wrote:

Gator TPK wrote:

. . .

I'm thoroughly impressed with your response times! I was outside and my iPhone "dinged" letting me know.

I was sitting here working on a revision of #B6 for this kind of situation. (it wasn't as complicated as I thought it would be). When convenient, please see if it's clearer now, and let me know.

' "Associate" a data-only volume (or a second OSX volume) ', is such a simple and clearer way to put it!

Though I should have seen my mistake and understood what you meant in the first place. 😝

Pondini wrote:

Gator TPK wrote:

Also, "sudo chflags -R nouchg " is still running. Would it be not advisable to run the "sudo rm -rf " command in another terminal window on the same package simultaneously?

Again, it sounds like it might be trouble, but wayyy out of my comfort zone!

I agree, I wasn't actually going to try it. That would be out of my comfort zone too! But that "chflags -R nouchg " is still running. I guess I have to wait on 2TB+ files.

Unless you said something like "Of course you can run lot's of Unix commands simutaniously in different terminals even if they're doing similar things!" 😝 😁

Gator TPK wrote:

. . .

Unless you said something like "Of course you can run lot's of Unix commands simutaniously in different terminals even if they're doing similar things!" 😝 😁

In theory, you should be able to. In practice, who knows? 😕

Pondini wrote:

Gator TPK wrote:

. . .

I hesitate to use the sudo command, unless someone else suggests it. Do you have any other suggestions on how to delete this pesky ".inProgress" file.

You might be able to use sudo rm -rf on it, but that's just a guess (I avoid Terminal if at all possible, and know just enough UNIX to get in trouble 😉 ).

It's been a while, and I've tried virtually everything I can to erase the ".inProgress" file. "Sudo rm -rf" just returned about 51 million characters (51 MB text file), every line being "rm: [file path]: Operation not permitted" OR "rm: [file path]: Directory not empty".

I even tried restarting from my other very clean and simple OS X partition, and magically those locked files with grayed out checked lock boxes in the "Get Info", I could uncheck! (without even undoing the lock at the bottom of the Info Box.) But I still can't delete the files. Chflags clearing still even gives me "Operation not permitted" on exactly every locked file, even though I can unlock it with the finder, (still can't delete the individual unlocked files singly).

What would happen if I tried turning on Time Machine with this second OS X partition I'm running now? Or if I use the associate command with this second OS X volume, and returned to my original main OS X partition? They're the same volumes Time Machine is backing up, though I'm not sure this would be wise?

Time Machine has been OFF since before my last post, and this OS X partition re-calculated the .inProgress file size. It's a ridiculous 2.69 TB. So, yes, smoke and mirrors as you say about the file sizes, it has 22 dated folder attempts within the package, not even a single unlocked file within there can be deleted. This iMac just spent ~7 hours last night deleting the ".inProgress file", it finally gave me 0 files to delete when asking for my password then ten seconds later I get the familiar : (very annoying)

The only thing I haven't tried is going to single user mode, but I haven't figured out how to mount the Time Machine volume from single user mode to rm (remove) the files (.inProgress package). I know how to mount the boot volume: "mount -uw / " to make it writable.

One thing that may be of relevance (that I hadn't mentioned), is that my main OS X partition is technically a Lion Server 10.7.2, but I haven't been using it as such since I experimented with it. This current (second OS X partition) is a regular Lion partition version 10.7.2+.

Any suggestions? Should I try turning Time Machine on again with the 2.69 TB .inProgress file remaining from either OS X partition? Or should I still attempt to delete the ".inProgress" package (using a yet unknown method)? Or might you know how to mount the time machine volume in single user mode? (I do know about running the "rc" program "sh /etc/rc ", to get more parts of OS X running without leaving single user mode and entering the GUI. I'm not even sure I need to though, just to "rm -rf" a package.)

Sounds to me like you've run afoul of the ACLs. If you're comfy with them (I'm not), you might be able to remove or revise them via chmod so you can delete the files.

Otherwise, your best bet may be to either start fresh on a new TM drive (you can always view and restore from them via the Browse.. option per #E2 in Time Machine - Troubleshooting), or erase the TM drive and start over.

Pondini wrote:

Sounds to me like you've run afoul of the ACLs. If you're comfy with them (I'm not), you might be able to remove or revise them via chmod so you can delete the files.

Otherwise, your best bet may be to either start fresh on a new TM drive (you can always view and restore from them via the Browse.. option per #E2 in Time Machine - Troubleshooting), or erase the TM drive and start over.

The Browse... option per #E2 is interesting (new to me!), I didn't look much into it yet.

Thanks for pointing me into looking at the ACLs. When I do "ls -le" to see the ACL on the ".inProgress" package, I get two ACLs:

__________

drwxr-xr-x@ 12 gatortpk staff 408 Dec 11 14:46 2011-12-04-193819.inProgress

0: group:everyone deny add_file,delete,add_subdirectory,writeattr,writeextattr,chown

1: C669022E-73EF-4507-9354-CDFB0BB4957C allow list,add_file,search,add_subdirectory,readattr,writeattr,readextattr,writeextat tr,readsecurity

__________

That second ACL isn't a group or a user! Is a Hex number ACL normal for a Time Machine package? ("2011-12-04-193819.inProgress" is the package I put in bold)

I finally found an old Time Machine HDD that still had a ".inProgress" package (from an old 12" PowerBook), and it only had the one ACL, the first as seen above starting with "0: group:everyone..." It's actually identical. So I suppose it is likely "normal".

Possibly This second: "1: C669022E-73EF-4507-9354-CDFB0BB4957C allow " ACL is getting in my way, even though it's a "allow" and not "deny" one?

Gator TPK wrote:

. . .

Otherwise, your best bet may be to either start fresh on a new TM drive (you can always view and restore from them via the Browse.. option per #E2 in Time Machine - Troubleshooting), or erase the TM drive and start over.

The Browse... option per #E2 is interesting (new to me!), I didn't look much into it yet.

I suspect you've figured it out, but I should have said "you can always view and restore from the old ones . . "

Thanks for pointing me into looking at the ACLs. When I do "ls -le" to see the ACL on the ".inProgress" package, I get two ACLs:

Sorry, I know ACLs exist; I know Time Machine puts very restrictive ones on completed backups, but that's about all I know (and I've never had or seen such problems trying to delete an inProgress package, either).

You're wayyyy out of my comfort zone here. 😕

I should probably refer you to the (free) BatChmod app, that I see the UNIX gurus talking about. Looks like it might help.

(One of these days, I suppose I should dip a toe in those waters . . . 😉 )

Pondini wrote:

Sounds to me like you've run afoul of the ACLs. If you're comfy with them (I'm not), you might be able to remove or revise them via chmod so you can delete the files.

Thank you! Thank you again for pointing me to those ACLs! I've done many other chmod commands before getting to this one. (I'm leaving mostly irrelevant info out)

I tried "chmod -R -N [folder path]" on one folder inside the ".inProgress" package first because I knew it was a smaller folder, and I didn't want to wait 7 hours to see if that chmod trick worked. And it did work! So now, I've been seeing what works and doesn't on smaller folders (and files).

One very interesting and consistent note is that in the Info Box of the files and folders that I can't delete in any way, an extra custom "Sharing & Permissions:" for everyone, that is what is keeping me from deleting the folders. I can't remove this privilege from the Finder, but a simple "chmod -R -N" works and that highlighted (below) custom privilege goes away. (No super-user required, I don't believe this makes sense, I'm allow to remove an ACL that blocks a "sudo rm -rf" command?) It looks like this:

I also noticed while deleting some subdirectories, that I would get that same annoying message "The operation can't be completed because some items had to be skipped. For each item, choose File > Get info, make sure "Locked" is deselected, ....". The only thing keeping me from deleting these annoying folders were locked files, even just one locked file will stop 500,000 from deleting. So "chflags -R nouchgs [folder path]" would unlock everything in the folder at once. Finally everything is deletable. I had tried the "chflags -R nouchgs [folder path]" two days ago, but it wouldn't work until I chmod and deleted the ACLs first.

I have lastly typed "chmod -R -N 2011-12-04-193819.inProgress" and it is deleting all the ridiculous ACLs. Though all the folders inside the package had the same ACLs, the "-R" recursively "-N" removed all the ACLs. I'm waiting for this to go through. Then I'll "chflags -R nouchgs [.inProgress package]". Then I'll delete the whole package from the Finder and maybe pray a little for 7 hours.

I'm sure there was an easier way, but just the deleting ACLs and unlocking files, then deleting takes time anyway. Perhaps I could have done "chmod -R a=w [.inProgress package]". Thus allowing everyone access to delete the package? But maybe the deny ACLs would have still stopped me, and the locked files too? I'm not doing anymore experimenting. At least I know my way around the terminal again, and learned a little more UNIX!

I'll keep you posted, and see if the ".inProgress" package file has been deleted! Then I have to remember why I've been doing this for two days now. Oh yes! To start Time Machine and see if it'll work again! 😝

Pondini wrote:

I should probably refer you to the (free) BatChmod app, that I see the UNIX gurus talking about. Looks like it might help.

(One of these days, I suppose I should dip a toe in those waters . . . 😉 )

I just wanted to update. (I just now looked at BatChmod, thanks for the tip. 🙂)

The terminal and chmod isn't so bad (and chflags -R nouchg for unlocking all files inside a directory), and I never worried about damaging data (as long as I was working within the package file and not using sudo). I didn't have to use a single "sudo" command, except to delete (sudo rm -rf) the empty ".inProgress" file (that command wouldn't work when it was full, as pointed out last earlier this week). I did have to end up deleting folders one (or up to six) at a time inside. I would run in to some unusual errors when doing too large a directory or the entire package, so it was like deleting parts of 22 hard drives, folder by folder till I cleaned out the whole package. I did all this from the Finder. The only time i used the terminal at the end, was to look at ACLs from the inside and to use 'chmod' and 'chflags -R nouchg' commands on the folders before deleting them from the Finder.

I deleted the empty ".inProgress" at 4:58 PM EST today. I rebooted into my main partition and started Time Machine around 5:05 PM, and it started backing up about an hour ago (around 9:15 PM), after saying something like "indexing backup" most of the time.

Unfortunately, it is backing up 675.7 GB of data (the full backup). I think the whole point I got to this post, was to avoid this. I was wondering, if I don't get any errors (and it doesn't fail 22 times building up a massive inProgress package) should I let it go? I have 1.53 TB space remaining on 3 TB. (the 2.69 TB package was about a real 710 GB)

The big question: Will I still have about a half empty TM HDD when it's all said and done because it'll delete the other older but identical files through seeing identical "hard links"?

Sorry if you have already addressed this on one of your web pages, I do remember you saying somewhere this is what may happen eventually, it'll look like a full backup but then turn out being an incremental (delta) backup.

Gator TPK wrote:

. . .

Unfortunately, it is backing up 675.7 GB of data (the full backup). I think the whole point I got to this post, was to avoid this. x

Did you do the "associatedisk" operations?

I was wondering, if I don't get any errors (and it doesn't fail 22 times building up a massive inProgress package) should I let it go? I have 1.53 TB space remaining on 3 TB.

There's no telling whether it will really back up the whole thing again (it's probably about as boggled as I am 😉 ), but it very well may.

Sometimes, it says it's going to back up everything, but seems to be proceeding extremely slowly, then when it's been a long time, but not much actually copied, it finishes, having copied only a fraction of what it predicted.

(the 2.69 TB package was about a real 710 GB)

That makes me wonder about the directory. 😕

The big question: Will I still have about a half empty TM HDD when it's all said and done because it'll delete the other older but identical files through seeing identical "hard links"?

No, if it actually copies everything, you'll have two sets of it all. (Usually, anyway -- you've had so many odd things happen, I don't think I should be predicting anything!).

Pondini wrote:

Gator TPK wrote:

. . .

Unfortunately, it is backing up 675.7 GB of data (the full backup). I think the whole point I got to this post, was to avoid this.

Did you do the "associatedisk" operations?

No, I didn't do the "associatedisk" operations, I just turned Time Machine back on just to see if it would resume delta backups. Now, I know that it's not. I think I'll let it go for now. I can always delete that fresh 675 GB full backup and then do "associatedisk", if I end up with only 855 GB free space remaining. I guess I just wanted to see that Time Machine is actually working again. Last time after I got the new internal HDD through Apple Care, it failed (22 times) every time within an hour. It's at 221 GB now, and seems to have indexed fine and doesn't have crazy ACL problems anymore.

Sometimes, it says it's going to back up everything, but seems to be proceeding extremely slowly, then when it's been a long time, but not much actually copied, it finishes, having copied only a fraction of what it predicted.

It appears to be taking exactly the right amount of free space. It says 1.31 TB of 3 TB at about 221 GB copied when it started out at 1.53 TB of 3 TB. But when it cleans up at the end I wonder if it'll pop up back to around 1.5 TB of 3 TB. I'll have to wait and see!

Pondini wrote:

Gator TPK wrote:

. . .

(the 2.69 TB package was about a real 710 GB)

That makes me wonder about the directory. 😕

Actually it makes perfect sense, the 2.69 TB calculation of the package is just like the Whole Time Machine Folder being 20 times larger than the HDD itself. There were 22 partial attempt to backup my iMac inside the ".inProgress" package, and if you add everything up, that's about right. There were "hard links" within the package. And every (22nd) duplicate file inside the package was an alias except for one.

(Sorry, I haven't figured out the intracacies of quoting.)