Server 3.2.1 (10.9.5) – Postgres won't start

Well, okay, things look better, but Profile Manager still isn't running. The only clue is the xpostgres.log; here are 50 lines. It repeats.

Mainly, it seems to be confused about the migrate/update status of the database. Postgres is running

postgres:dataDirHasBeenInitialized = yes

postgres:PG_VERSION = "9.3.4"

postgres:dataDir = "/Library/Server/PostgreSQL/Data"

postgres:postgresIsResponding = yes

postgres:dataDirIsDirectory = yes

postgres:PGserverVersion = 90304

postgres:dataDirExists = yes

postgres:setStateVersion = 1

postgres:state = "RUNNING"

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/python/failure.py", line 382, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

File "/Applications/Server.app/Contents/ServerRoot/usr/bin/xpostgres", line 2033, in main

result = yield xpg.do_everything(argv, environ)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/internet/defer.py", line 1043, in _inlineCallbacks

result = result.throwExceptionIntoGenerator(g)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/python/failure.py", line 382, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

File "/Applications/Server.app/Contents/ServerRoot/usr/bin/xpostgres", line 1094, in do_everything

yield self.do_restore()

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/internet/defer.py", line 1045, in _inlineCallbacks

result = g.send(result)

File "/Applications/Server.app/Contents/ServerRoot/usr/bin/xpostgres", line 1028, in do_restore

did_upgrade = self.upgrade_cluster_if_needed()

File "/Applications/Server.app/Contents/ServerRoot/usr/bin/xpostgres", line 1455, in upgrade_cluster_if_needed

os.rename(self.data_directory, orig_data_dir)

exceptions.OSError: [Errno 66] Directory not empty

2014-09-29 20:32:49 XPG.1053: WARNING: parameter "unix_socket_directory" was specified but is no longer valid. Using the specified value for now, but please use "unix_socket_directories" in the future.

2014-09-29 20:32:49 XPG.1053: Doing restore.

2014-09-29 20:32:49 XPG.1053: Spawning... ('/usr/bin/tar', '-xz', '-f', '/Library/Server/ProfileManager/Config/ServiceData/Data/backup/base_backup/base _complete.tar.gz', '-C', '/Library/Server/ProfileManager/Config/ServiceData/Data/PostgreSQL')

2014-09-29 20:32:50 XPG.1053: Preparing to upgrade outdated database cluster.

Traceback (most recent call last):

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/internet/defer.py", line 397, in errback

self._startRunCallbacks(fail)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/internet/defer.py", line 464, in _startRunCallbacks

self._runCallbacks()

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/internet/defer.py", line 551, in _runCallbacks

current.result = callback(current.result, *args, **kw)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/internet/defer.py", line 1101, in gotResult

_inlineCallbacks(r, g, deferred)

--- <exception caught here> ---

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/internet/defer.py", line 1043, in _inlineCallbacks

result = result.throwExceptionIntoGenerator(g)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/python/failure.py", line 382, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

File "/Applications/Server.app/Contents/ServerRoot/usr/bin/xpostgres", line 2033, in main

result = yield xpg.do_everything(argv, environ)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/internet/defer.py", line 1043, in _inlineCallbacks

result = result.throwExceptionIntoGenerator(g)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/python/failure.py", line 382, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

File "/Applications/Server.app/Contents/ServerRoot/usr/bin/xpostgres", line 1094, in do_everything

yield self.do_restore()

File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twi sted/internet/defer.py", line 1045, in _inlineCallbacks

result = g.send(result)

File "/Applications/Server.app/Contents/ServerRoot/usr/bin/xpostgres", line 1028, in do_restore

did_upgrade = self.upgrade_cluster_if_needed()

File "/Applications/Server.app/Contents/ServerRoot/usr/bin/xpostgres", line 1455, in upgrade_cluster_if_needed

os.rename(self.data_directory, orig_data_dir)

exceptions.OSError: [Errno 66] Directory not empty

Hi Jeff,

Thanks a lot!!!! This worked within 30 seconds. Couldn't even get a cup of coffee 😟

Thanks Rusty for the script and René for the /tmp tip. Postgres seems happy again on my server.

Hi Jeff,

Have not done the upgrade yet, glad I looked before I leaped, lol.

I do use the topicdesk.com setup of Roundcube. Before I do the update can I do a sqlDB backup, then update 10.9.5, server 3.2.1, run your script, then restore sqlDB and OD? I know to do OD image already prior to backup.

Anyone else want to write full set of directions to upgrade to 3.2.1, or just clue me in on other pitfalls? I do not use Profiles btw, just email(SMTP, POP, IMAP), topicdesk webmail setup, AFP, SMB, Web, OD, Mobile Homes for backup.

Mike

PS anyone with Enterprise Server Support get info on a release fix?

I actually did need to run Rusty's original commands to create the Data directory:

1. mkdir -p /Library/Server/PostgreSQL/Data
2. chown -R _postgres:_postgres /Library/Server/PostgreSQL/Data
3. chmod 700 /Library/Server/PostgreSQL/Data

Without doing that, the initdb command would fail saying:

creating directory /Library/Server/PostgreSQL/Data ... initdb: could not create directory "/Library/Server/PostgreSQL/Data": Permission denied

I'm not sure why. But, once I created the directory as per at the above commands, the rest worked. I'm up and running again.

Thanks very much for all the help everyone.

(by the way, I originally just created the Data directory myself and forgot to set the group to "_postgres". The group was "wheel," and when trying to start postgres I'd get "CANNOT_START_SERVICE_TIMEOUT_ERR", the same timeout others are seeing. Check the folder attributes to make sure both user and group are set to "_postgres" )

I just wanted to add a link to another shell script that fixed this issue for me. It's written by somebody named Jeff Johnson with the specific intent to repair postgres for Roundcube webmail as installed by the excellent version provided by topicdesk but should work fairly well as a general fix.

http://downloads.topicdesk.com/installers/PostgresRoundcubeFix.tgz

Line 16 gives me the following error

"Old cluster data and binary directories are from different major versions."

Same message in the postgres log files. I find it weird that none else here has posten the same error message, any idea?

EDIT: the reason i find it weird is because I come from a fresh install of the previous version of OS X Server (3.1.2??), so I would guess I had the latest previous version of postgres installed, so why the "different major versions"?

Awesome work Jeff. As usual you are the man when it comes to Roundcube. Love your work! Saved me a lot of time and effort today. Really appreciate it.🙂

I am having this same issue as well. This is what happens when I run the second line of the upgrade script.

sudo -u _postgres /Applications/Server.app/Contents/ServerRoot/usr/bin/pg_upgrade -b /Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2 -B /Applications/Server.app/Contents/ServerRoot/usr/bin -d /Library/Server/PostgreSQL/DataOld -D /Library/Server/PostgreSQL/Data

Performing Consistency Checks

-----------------------------

Checking cluster versions

Old cluster data and binary directories are from different major versions.

Failure, exiting

Have you run Jeff's fix file?

Yup tried it a few times. I think I have fixed the issue.

It seemed to be multiple issues causing the upgrade failure:

One was the updated plist file which I updated from rusty's recent post. That allowed my postgres install to run which had been part of the issue blocking Jeff's fix from working properly.

I ran the fix and webmail is working again.

I have not been able to successfully migrate over the old data DB folder that still produces the error but thats ok.

After inspecting the /Library/Server/Postgres folder, I noticed that the Data folder and the OldData folder both contained files, and a file named PG_VERSION Said 9.3 on both. It looks like postgres was migrated after all, so I skipped line 16 and continued onwards, and now "serveradmin status postgres" said "RUNNING".

I can assume that postgres is working now but I still have problems with the following:

Calendar (won't start, just "STARTING")

Contacts (won't start, just "STARTING")

Messages (Users can't log inn)

Profile Manager "Trust Profile" and Profile Manager "Settings for everybody" (both are unchecked and won't "check").

In addition my server is now running on constant 5-10% CPU load, up from 1-2%

This is shait!

Are you using the Web Service in Server app? I have one website using the Web Service and other Ruby on Rails applications using the built-in PostgreSQL database. The only time I can start PostgreSQL is if I stop the web service and start it again. Otherwise I get the postgres:error = "CANNOT_START_SERVICE_TIMEOUT_ERR". If I use your suggestion and rebuild the database do you know if the Web Service in Server app will continue to work?

Which socket directory? Please clarify.

I'm seeing every error people are posting about and none of the fixes are working. The service timeout message doesn't actually prevent postgres from starting.. it just runs around in circles. All of my services work fine EXCEPT for Profile Manager, which I happen to need. At least a full restart clears out the Device Management so that isn't on constant restart.

This is what startup fail looks like, minus the Device Manager config:

10/12/14 12:24:51.948 PM collabd[3397]: [CSNotificationService.m:96 1e81000 +220ms] XXX: Woke up, but didn't receive anything. This shouldn't happen, possible bug.

10/12/14 12:25:19.972 PM servermgrd[3343]: -[SocketRequest sendRawDataRequestToDevicemgrd:]: EXCEPTION: -[SocketRequest sendRawDataRequestToDevicemgrd:] (/SourceCache/RemoteDeviceManagement/RemoteDeviceManagement-875.25/Compiled/Fra mework-Base/Support/SocketRequest.m:89): "'(!_async && _bufferedInput.length == 0)'"

10/12/14 12:25:19.978 PM Server[3074]: Error starting Profile Manager service

10/12/14 12:25:19.978 PM Server[3074]: Error: The server '127.0.0.1' reported an error while processing a command of type: 'readSettings' in plug-in: 'servermgr_devicemgr'. Error: Error Domain=servermgr_devicemgr Code=1 "Devicemgrd is not running" UserInfo=0x60000107ac00 {NSLocalizedDescription=Devicemgrd is not running}

What socket?

I bypassed any problems with these commands:

#DUMP DB BEFORE UPDATE SERVER.APP. USE TERMINAL

sudo /Applications/Server.app/Contents/ServerRoot/usr/bin/pg_dumpall -h /Library/Server/ProfileManager/Config/var/PostgreSQL/ -U _devicemgr -c -f ~/Desktop/profilemanager_dumpall.pgdump

#DELETE /Library/Server/ProfileManager

sudo rm -R /Library/Server/ProfileManager

#UPDATE SERVER.APP, CLOSE IT AND RESTORE USING TERMINAL

sudo serveradmin stop devicemgr

sudo /Applications/Server.app/Contents/ServerRoot/usr/share/devicemgr/backend/wipeDB .sh

sudo dropdb -h "/Library/Server/ProfileManager/Config/var/PostgreSQL" -U _devicemgr devicemgr_v2m0

sudo createdb -h "/Library/Server/ProfileManager/Config/var/PostgreSQL" -U _devicemgr devicemgr_v2m0

sudo psql -h "/Library/Server/ProfileManager/Config/var/PostgreSQL" -U _devicemgr -d devicemgr_v2m0 -f ~/Desktop/profilemanager_dumpall.pgdump

sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "UPDATE auto_join_profiles SET usage_log = NULL"

sudo /Applications/Server.app/Contents/ServerRoot/usr/share/devicemgr/config/migrate DB

sudo serveradmin start devicemgr