Moving home directory

Thanks, but actually, the procedure described in that article did work - after unlinking the OneDrive app, as that was indeed what was preventing me from copying my home directory to the external drive.

After doing that, I logged out, logged into my alternative admin account, and used that to change my account's home directory to the one on the external drive. Then restarted the machine.

This all worked fine and appears to function normally... *except* that now I cannot relink the OneDrive app. OneDrive refuses to create its local directory on a removable drive, and if I try to make it use a directory owned by my account on the internal drive, I get the dreaded "We can't sync your One Drive - ********* University folder" error, which has been reported dozens of times on the Microsoft support forum and appears to have no reliable solution.

So, partial success, but only partial.

An update and a followup question. I did successfully move my home directory to the external drive. But I found that in that configuration, it is impossible to get OneDrive to work. Only the directory structure of the OneDrive filesystem is visible, either in the Finder or via the command line (i.e., ls). No files at all are visible that need to be accessed from the cloud - files can be placed there, and they will sync to the cloud, but if deleted and then restored from the OneDrive Recycle Bin, they will not be visible locally, only the directories they're in. And it makes no difference whether I designate a directory in my home directory or on the internal drive as my OneDrive folder - the behavior is exactly the same. The critical factor is the location of the home directory - as far as I can tell, OneDrive simply does not work if it is anywhere but on the internal drive. (and yes, I formatted the external drive as APFS... still does not work)

On the other hand, OneDrive DOES work, seamlessly as far as I can tell so far, if my home directory is on the internal drive. So I have moved it back there and am looking for alternate solutions to my space problem. Basically, I'm back to moving specific subdirectories within my home directory (e.g. Documents) onto the external drive and symlinking from them to my home directory. Here's what I have done:

1. Log into my alternate admin account.
2. Clone the Documents directory in my home directory to the external drive (sudo ditto /Users/<user>/Documents /Volumes/Users/<user>/Documents
3. From there, delete the Documents directory in my home directory. (sudo rm -rf ~<user>/Documents)
4. Then symlink from the cloned Documents directory on the external drive to my home directory. (sudo ln -s /Volumes/Users/<user>/Documents /Users/<user>)

This seems to work and frees up over 100 GB on my internal drive. I can get even more back if I do similarly with ~/Downloads and ~/Pictures. But some wise people here have advised me that the only safe solution to what I want to do is to move my entire user account to the external drive. Well, that has proven unworkable because I need OneDrive. So exactly what is my current solution (offloading only specific directories) going to break?

Barney-15E wrote:

Thanks, but actually, the procedure described in that article did work

…

After doing that, I logged out, logged into my alternative admin account, and used that to change my account's home directory to the one on the external drive. Then restarted the machine.

Then you didn’t follow that article. You actually did it correctly because you logged out of your account and logged into another account to make the changes. Nothing in that article says to do that. Making the changes from the logged in account causes all kinds of problems.

Correct, but I did make one critical mistake which, according to @etresoft, could cause inconsistencies that only show up much later down the line: namely, that I did the actual copying while logged into my account. I only signed into the alternate admin account to change my account's home directory in Users and Groups. As I said, it did appear to work (except possibly for the OneDrive issue which could have been unrelated), but... I was leery of what I had done so I did a clean reinstall and then used Migration Assistant to restore my files.

And guess what? man is still trying to call /bin/more and thus I can't read any man pages on my Mac.

I think you said that man works normally for you. Which version of MacOS are you running? I'm trying to determine if this might be a bug in 13.1. The other possibility is that Migration Assistant overwrote something when restoring my files. I did tell it to restore system files as well as my own, and the last safe backup that I trust was from when I was running Big Sur on my previous Mac (now defunct).

don’t know which would be worse, copying the files with them possibly open or fixing all of the permission issues caused by coping from another account. ditto may provide the ability to preserve the permissions which then would be the best way.

I will try to move my account later today, using ditto to do the copy whilst logged into my alternate admin account. Thus far I have not relinked OneDrive, and in fact the directory that was causing difficulties (~/Library/CloudStorage) does not currently exist. I'll let the forum know how that goes.

So it turns out that enableOwnership (assuming that's the same bit that one sets in the Get Info dialog) is indeed the crucial thing needed to make OneDrive work on an external volume. Now, with my home directory back on the external drive, it works fine! It's not clear if the reason it didn't work earlier was that I didn't own any of the space in my home directory, or whether it was directly because the enableOwnership bit wasn't set on the volume holding the cache. I could do more experimenting to sort this out, but I think I will just let things be for now and say...

Problem solved! :)

Thanks especially to @BobHarris here for the idea that led to the solution.

First, thank you for an *informative* reply without condescension or attitude.

BobHarris wrote:

/Users/username and /Users/username/Library would be best kept on the boot disk, as that way the key account files would always be available. Even if the external drive is missing.

Yes, I've considered that, but I have an alternative account with admin privilege that I can log into if the external disk is unavailable. More crucial is my account's data - but as you say, as long as I have good and frequent backups I'm not too worried.

All the other Folders could be moved to an external drive and replaced with symbolic links.

Symbolic Links are a file system abstraction at the lowest level, and unless the program or app does a stat() system call and looks at the mode bits, it will not know the symlink exists.

Yes exactly, which is why I was surprised to learn that MacOS treats directories like ~/Documents in such a way that replacing them with symlinks could break things. But I accept that I have no idea what Apple might do in the future, so I'm content to leave them where they are - as long as I can offload their CONTENT without risk of losing it.

This does not mean that sometime in the future you will not run into an app that is being too clever with respect to symbolic links, or that macOS will not create a magical file system attribute that it decides to put on key $HOME folders, that a symbolic link cannot hold, or that can only be applied by calling a new system call that does not have a command line equivalent.

And they already have associated ACLs, right? So we shouldn't be too surprised if they do that. Apple treats ~/Documents, ~/Downloads, ~/Pictures, and other user directories in special ways. This is why I want to know what is SUPPORTED and will not break in the future.

If it is not working, then you are being too aggressive in how much you are moving to external storage, and at login, the needed files to complete the login are missing.

Moving the entire home directory to external storage actually works fine, EXCEPT that it breaks OneDrive. As I said to @etresoft, I would chuck it if I could. But I cannot work from home without it.

In the future, I strongly suggest you size the internal storage on your next Mac so it can hold ALL your current (at the time of purchase) and additional storage needs during the life of the next Mac. It makes life much simpler not to need to play external storage games.

Sure, and ideally I would have. I had not been aware of space issues (I think my system disk was 50% full) when my old MacBook Pro crashed and could not be rebooted. I could not even reinstall the OS without errors, so reluctantly I ordered a new machine. Yes, it would have been more prudent to choose a larger internal drive (I doubled the RAM), but from what I had read about external SSD drives and looking at the price delta, I decided I would go that route if it ever proved necessary... and it did, and sooner than I expected.

Also the internal storage is going to be faster then most external attached storage.

I expected external to be slower, but I was pleasantly surprised: I cannot notice any difference.

PS. I’ve been a Unix file system developer on several different Unix implementations for the past 28 years. I am very well aware of how Unix file systems treat Symbolic Links (never for Apple; but I’ve used Macs since ’88 and paid attention to its different file system implementations over the years).

I owned a Mac II back in the '80s before they went to MacOS, but then left Apple world for many years and used Unix, and later Linux, systems almost exclusively. My first exposure to Unix was on a Sun 3 as a grad student. As a postdoc I was a sysadmin for a cluster that included machines running IRIX, Linux, and a couple of DEC machines running Ultrix thrown into the mix. I've studied Unix internals to some extent but never developed any software that made use of low-level calls. And I have no knowledge of Apple's proprietary file systems like HFS+ and APFS. I'm not a hacker - I'm just a user with plenty of Unix / Linux experience who is finding this new-to-me flavor a bit intimidating. Thank you for the information.

I just tried copying my home directory to the external drive while logged into the alternate admin account. The beginning of the output is below. It appears this cannot be done easily...

%sudo ditto /Users/lise /Volumes/Users/lise

Password:

ditto: /Users/lise/./Library/Application Support/MobileSync: Operation not permitted

ditto: /Users/lise/./Library/Application Support/CallHistoryTransactions: Operation not permitted

ditto: /Users/lise/./Library/Application Support/CloudDocs/session/db: Operation not permitted

ditto: /Users/lise/./Library/Application Support/com.apple.sharedfilelist: Operation not permitted

ditto: /Users/lise/./Library/Application Support/Knowledge: Operation not permitted

ditto: /Users/lise/./Library/Application Support/com.apple.TCC: Operation not permitted

ditto: /Users/lise/./Library/Application Support/FileProvider: Operation not permitted

ditto: /Users/lise/./Library/Application Support/FaceTime: Operation not permitted

ditto: /Users/lise/./Library/Application Support/com.apple.avfoundation/Frecents: Operation not permitted

ditto: /Users/lise/./Library/Application Support/CallHistoryDB: Operation not permitted

I see these files are all in ~/Library, and I did get similar errors from somewhere within that directory the first time I tried to copy my home directory in the Finder (this was yesterday, while logged iinto my actual account), with no indication of the offending files.

Is there any way to troubleshoot these "Operation not permitted" errors? It appears that I do not even have permission to list the contents of at least one of these directories (~/Library/Application Support/CallHistoryDB), even though ls reveals it to be a simple mode 755 without any ACL or extended attributes:

drwxr-xr-x 8 lise staff 256 Dec 24 06:33 CallHistoryDB

Yet,

% ls -al ~/Library/Application\ Support/CallHistoryDB

total 0

ls: /Users/lise/Library/Application Support/CallHistoryDB: Operation not permitted

The Finder does not even display this directory.

etresoft wrote:

Several of the folders in your home directory, including Documents and Downloads, are baked into the operating system security at a low level. No one, not me, and not even anyone at Apple, has any idea what will happen if you try to hack those up with symlinks. That was always a bad idea and will be an even worse idea going forward. It's like dropping a time bomb in your home directory. It's going to go off one day and irretrievably wreck your documents - maybe not this year, or the next, but it will be catastrophic.

That's what I'm afraid of. But at this point I don't see any other solution. Yes, I should have sprung for a larger internal drive, but I had no idea Ventura would take up so much more space than Big Sur. And it's water under the bridge at this point.

There is no problem with changing your home directory. /Users is no different than any other path. Anything can be a home directory path as long as it is configured in the proper way. If OneDrive can't handle that, then it's a bug in OneDrive.

Are you sure? Microsoft said earlier this year that whatever limitations exist in their implementation as far as syncing external drives with its cloud are actually imposed by Apple's File Provider extension, which OneDrive relies on. Any chance this could be true?

Downloads I think can be changed in Safari -> Preferences -> General -> File Download location

https://www.alphr.com/change-downloads-folder-mac/

Yes, that's what I did - actually the equivalent option in Firefox, as I do not use Safari.

You can move your Photos library to a different location, and it is fully supported by macOS

Move your Photos library to save space on your Mac - Apple Support

The Music library can also be moved

Change where your music files are stored on Mac - Apple Support

The same for the Movies library

Move your iMovie for Mac library - Apple Support

Thanks - but those areas are much smaller in my case than Documents and Downloads, so there is little point in moving them.

That is a discussion for Microsoft. Or you should look for a different file sharing service. I have zero OneDrive experience, so I cannot be much help there.

A Microsoft agent finally got back to me today (on their support forum) with instructions for me to relay to our IT department to open a ticket with them, so if there is an answer to that issue, that's where I'll find it. I have a sneaking suspicion though, that the problem is a limitation in Apple's AFP which OneDrive now uses to sync between local storage and their cloud server. We'll see. In any case, the choice of service isn't mine - it's my employer's. (If it were my choice, I definitely wouldn't be using a Microsoft product! ;))

With respect to permissions and ACLs, you have to 'enableOwnership' on that device, or it does not enforce any ownership and permissions (including ACLs) on the files stored on the external device.

See "man diskutil" and look for the 'enableOwnership' argument.

Ok, thanks - but I have a question about that. Is that the same thing as the "Ignore ownership on this volume" option that can be set, if your account has admin privilege, in the Get Info dialog for a volume? I was indeed not careful enough about that to begin with and ended up with all of my files owned by "unknown:unknown" when I first transferred my files to the external volume. In fact I did not discover it until after I had moved my home directory back to the startup volume, when I found that all of my files had the wrong ownership and had to chown them back manually. The ACLs and extended attributes appear to have been unaffected - but the ownership was definitely wrong, and I didn't realize it until I did an ls on my home directory from the alternate account and found that everything there seemed to be owned (even when using -n) by the UID running the command. It took "sudo ls -an" to figure out what was wrong.

This does NOT happen if that box is unchecked, so I wonder if that's the same thing as enableOwnership. (It does sound like it, based on the diskutil man page.)

However, if it is working to your satisfaction, then leave 'enableOwnership' alone, or at least be prepared to use 'disableOwnership' if it does not work to your satisfaction.

As mentioned above, I have moved my home directory back to the startup drive for the time being because it seems OneDrive will not sync correctly otherwise. I have not yet tried OneDrive on the external drive with ownership enabled on that volume though - that might be something to try.

As to how you moved your home directory. Did you replace /Users/username with a symbolic link?

Definitely not.

Or did you use System Preferences -> Users & Groups -> unlock padlock in lower left corner of window -> Control-Click on your account -> Advanced Options -> Home Directory

That is what I did.

I mention the User & Groups Advanced Options, because maybe OneDrive will behave if you make the change there are do not use symbolic links.

No. The only thing left to try there is with volume ownership enabled. Otherwise, it is a bug either in MS's code, or in Apple's.

Thanks again for a helpful reply.

Then I misunderstood. I thought it was only the change of home directory associated with the account that had to be done from a different account. The home directory could not be copied, even from the alternate admin account, without first unlinking the OneDrive app. I still got the "has to be downloaded" error and only partial copies. But I didn't realize I needed to be logged into a different account to copy the home directory itself.

So do you think my safest bet is just to wipe the internal drive, reinstall the OS and then restore my account from a backup, before redoing the home directory move the right way? Right now I don't see any inconsistencies, and the OneDrive problem is likely something unrelated. But if this could cause problems way down the line, then maybe that's what I should do.