How to Migrate, Rebuild, and Fix OS X Server

I upgraded my server to new hardware, hand-migrated all its data, and fixed Profile Manager’s “500 Internal Server Error” problem. My experience has been the same as the Ars commenter who wrote that OS X Server is “extremely fragile, and when it breaks it breaks severely and inscrutably.”

The online documentation to rebuild and/or fix OS X Server is sparse and inadequate, so I’ve posted my ultimately successful steps here.

I believe that these steps are nearly all necessary, and the process is extremely fragile—a single mistake can break the entire setup, so proceed deliberately and carefully. I strongly recommend cloning a scratch copy of your server and verifying that you can do this at least TWICE while booted into the scratch version before performing any irretrievable destructive actions on your server’s actual boot partition. The basic strategy is to backup all data, not as archives which are UUID sensitive, but as data exports that can be imported after the rebuild. The magic steps to rebuild Profile Manager AND Open Directory involve destroying the OD Master AND running the wipeDB command AND destroying the LDAP server, then using Server.app and Workgroup Manager to rebuild everything from the ground up.

What you need:

(1) Purchase Carbon Copy Cloner (CCC)

(2) Purchase a disk toaster (any external HD), partitional with space for at least at least two bootable backups, one for scratch

(3) Download latest version of Workgroup Manager

Initial Migration [Skip if You’re Not Migrating from old hardware]

(1) Source partition: Start with a full bootable CCC clone with a Recovery Partition, AND a full Time Machine backup.

(2) Destination drive: Erase the Destination drive [DON’ MAKE A MISTAKE]

<http://www.cnet.com/news/tackle-stubborn-disk-partitioning-in-os-x/>

<http://www.ernieflores.net/unix/deleting-and-merging-a-partition-with-diskutil-f rom-the-command-line-osx/>

If the partition is encrypted [fast — avoid decryption]:

Disk Utility> Unmount Disk

diskutil list

sudo fdisk -i -a hfs /dev/disk0 # Use the correct disk number!!

The disk you inserted was not readable by this computer. Initialize…

(3) Disk Utility>Partition> 1 Partition, Server HD, Mac OS Extended (Journaled)

Options … GUID Partition Table

(4) Carbon Copy Cloner>Disk Center>Recovery HD>Create a Recovery partition for this volume…

(5) The presence of Server.app breaks Migration Assistant, so create a SECOND, scratch bootable clone of the Source partition, boot into it, delete server.app from the SECOND clone, then

(6) Reboot into Recovery partition (Command-R boot) on the Destination partition and restore OS

(7) Migration Assistant from from SECOND bootable clone. If you use a clone with Server.app, it WILL NOT WORK. https://discussions.apple.com/thread/4157337?answerId=22868828022#22868828022

Migrate Applications and possible Users ONLY. Do not migrate other data. If your new hardware is a small 256 GB SSD, you will probably have to migrate User data by hand onto an external drive, then use System Preferences>Users & Groups> Unlock, Right-Click on User > Advanced Options… > Home directory to locate user home directories off the small SSD drive.

(8) Also, the absence of /Users/Shared will break iTunes (“severely and inscrutably”), so use a symbolic link to get the possible large Shared folder onto a large external HD:

Terminal> $ sudo rm -fr /Users/Shared # SSD limit — put users on external HD

Terminal> $ sudo ln -s /Volumes/Macintosh\ HD/Users/Shared /Users/Shared

Terminal> $ sudo chmod 1777 /Users/Shared/

(9) For Server, you must also make sure the hostname is your FQDN. I forgot how I did this, but Old Lion Server advice still applies before installing Server.app <https://discussions.apple.com/thread/3565475?answerId=17005559022#17005559022>

Terminal> $ sudo scutil --set HostName server.domain.com

Rebuild and/or fix Profile Manager and Open Directory

(Test at least twice on a scratch partition to confirm that this works with your setup.)

Backup target partition:

(1) Backup all user data for mail/cal/contacts for each account:

(1a) Mail> Select one or more mailboxes, then choose Mailbox > Export Mailbox

(1b) Calendar> Click the calendar’s name, File > Export > Export

(1c) Contacts> Command-A to select all contacts, File > Export > Export vCard…

(2) Workgroup Manager> Select all network accounts EXCEPT diradmin, then Server>Export…

(3) Server.app>Open Directory Archive master

(4) ccc_preflight.sh for odbackup, pg_dumpall, etc.

(5) Server.app> Turn off all incoming services: Mail etc.

(6) CCC> Create a bootable clone backup and bootable clone scratch

Scratch partition for testing (boot into it):

(1) Finder> Delete Server.app and Empty Trash;

(2) After screen “Server app removal detected.”, reassign server DNS to router

(3) Keychain Access> System>My Certificates> Back up all FQDN Certificates including “Open Directory Certificate Authority,” “IntermediateCA_FQDN_1,” “Server Fallback SSL Certificate” including private keys beneath triangle toggle

(4) Keychain Access> System>My Certificates> Delete all FQDN/other mentioned certs from all keychains

(5) App Store> Install Server.app

(6) Launch Server.app

(7) Server.app> Turn on DNS

(8) System Preferences>Network>Cconfigure network to get DNS from 127.0.0.1; relaunch Server.app

(9) Server.app>Open Directory> Destroy OD Master (– button)

(10) Quit Server.app

(11) Terminal> sudo /Applications/Server.app/Contents/ServerRoot/usr/share/devicemgr/backend/wipeDB .sh

(12) Terminal> sudo slapconfig -destroyldapserver

(13) Launch Server.app>Profile Manager; If on, turn PM off and restart Server.app

(14) Server.app>PM> If PM off then Configure... Make sure to use certificates created (also watch keychain)

(15) Server.app>PM> Button to sign config profiles

(16) Server.app>PM> Turn on PM

(17) Work Manager>Server>Import… backed up network accounts; quit WGM

(18) Server.app>Users>Local Network Users> Add all new network accounts to Workgroup and reset all passwords

(19) Server.app>Certificates> Secure services using the FQDN, except possibly for port 80 websites

(20) Safari> Reset Safari

(21) Safari> https://FQDN/ Log into profile manager with newly created pmadmin account (diradmin logins borked in Server.app)

(22) Install Trust Profile, then enroll device (Server is the device). This should work, and all certs should be verified

Test user data on local (administrator) account:

(1) Log into local account on server

(2) Keychain Access> System>My Certificates> Back up all FQDN Certificates including “Open Directory Certificate Authority,” “IntermediateCA_FQDN_1,” “Server Fallback SSL Certificate” including private keys beneath triangle toggle

(3) Keychain Access>Login Keychain> del all FQDN certs from LOGIN keychain

(4) sudo rm -fr ~/Library/Application\ Support/Certificate\ Authority

(5) System Preferences>Internet Accounts> Delete all accounts corresponding to previous OD Master, actually all Mail accounts

(6) Server.app>Mail Turn on Mail service

(7) Mail> Delete any old FQDN SMTP servers

(8) Mail>Preferences… Add Mail account, use FQDN for mail and smtp servers

Some server-specific tweaks:

Postfix aliases:

Terminal> $ sudo serveradmin set mail:postfix:salias_maps = “hash:/Library/Server/Mail/Config/postfix/aliases”

Terminal> $ sudo postalias hash:/Library/Server/Mail/Config/postfix/aliases

Terminal> $ sudo newaliases

For aliases with the same username as local accounts:

Server.app>Users>All Users>Click on username, Gear>Edit Access to Services…

OR

Server.app>View>Show System Accounts

Server.app>Groups>com.apple.access_mail> Double-click, add local accounts as members

PostgreSQL:

Terminal> $ sudo serveradmin start postgres

# pg_hba.conf in directory /Library/Server/PostgreSQL/Data

Change jabber to use the TLD, e.g. user@domain.org:

(1) Terminal> $ cd /Library/Server/Messages/Config/jabberd

(2) Terminal> $ sudo cp sm.xml sm.xml.orig

(3) Terminal> $ sudo vi sm.xml : <id>FQDN</id> —> <id>TLD<id/>, e.g. <id>domainname.com</id>

(4) Server.app>Messages> Restart service

Macports:

# on (old) Source partition

http://trac.macports.org/wiki/Migration

Terminal> $ port -qv installed > myports.txt

Observed issues/bugs:

Profile Manager FAILS to create an OD master and shows a large yellow triangle with a message saying that an OD master was created but “an error occurred.” You have to start over completely.

From an existing local account with the Mail app, Mail could not verify my server’s identity. The trust chain showed the OLD server certificates THAT DO NOT EXIST ANYWHERE IN ANY KEYCHAIN. Make sure that you're securing your services with the latest certificate in Server.app>Certificates.

New set of certificates (CA, Intermediate, Code Signing), but Profile Manager Enroll Device still returns “500 internal server error”. Then the newly created CA and Intermediate certificates were deleted from my System keychain, presumably by Server.app. You have to start over completely.

No certificate creation, Server.app>PM certificate creation process goes into an infinite loop and the “Next” button while entering my certification organization and contact information. You have to start over completely.