pwinkeler

Q: caldavd failing to start after upgrade to Server 3.2.1

This problem, caldavd not starting after upgrading to Server 3.2.1, is a variation on the theme of PostgresQL not starting.  It too boils down to a postgres database not having been updated properly.  Below are the steps needed to fix this problem:

sudo su -

# Change to the directory with the postgres database for the calendars and contacts

cd /Library/Server/Calendar\ and\ Contacts/Data/Database.xpg

# Disable caldavd for now

launchctl unload -w /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/org.c alendarserver.calendarserver.plist

# move the broken database out of the way and build a new empty one

mv cluster.pg broken-cluster.pg

mkdir cluster.pg && chmod 700 cluster.pg && chown _calendar:_calendar cluster.pg

sudo -u _calendar /Applications/Server.app/Contents/ServerRoot/usr/bin/initdb "/Library/Server/Calendar and Contacts/Data/Database.xpg/cluster.pg" -E utf8 --lc-ctype=C

# Finish the upgrade job that never completed by explicitly invoking the previous version of postgres:

sudo -u _calendar /Applications/Server.app/Contents/ServerRoot/usr/bin/pg_upgrade -b /Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2 -B /Applications/Server.app/Contents/ServerRoot/usr/bin -d /Library/Server/Calendar\ and\ Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade/ -D /Library/Server/Calendar\ and\ Contacts/Data/Database.xpg/cluster.pg

# Mark the newly upgraded database as no longer in need of an upgrade

sudo -u _calendar touch cluster.pg/.NoRestoreNeeded

# Start caldavd back up

launchctl load -w /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/org.c alendarserver.calendarserver.plist

And everything should be the way it should have been in the first place!

Mac mini, OS X Mavericks (10.9.5), 16GB Ram; 500GB HD

Posted on Oct 19, 2014 11:46 AM

Close

Q: caldavd failing to start after upgrade to Server 3.2.1

  • All replies
  • Helpful answers

  • by Linc Davis,

    Linc Davis Linc Davis Oct 20, 2014 10:07 PM in response to pwinkeler
    Level 10 (207,926 points)
    Applications
    Oct 20, 2014 10:07 PM in response to pwinkeler

    Excellent. I also had to disable the org.calendarserver.agent daemon in the same way.

  • by Captain_Spaulding,Solvedanswer

    Captain_Spaulding Captain_Spaulding Oct 23, 2014 10:25 AM in response to pwinkeler
    Level 1 (10 points)
    Oct 23, 2014 10:25 AM in response to pwinkeler

    You nailed it.  The failed upgrade had resulted in all the contacts disappearing from the carddav service on Mavericks after I upgraded to Server 3.2.2 (I had skipped 3.2.1).  The only adjustment I had to make to restore my database was:

     

    1) Change the paths for the DB locations to point to the external volume where I keep my service data

    2) add " --lc-collate=C" to the initdb line as I otherwise I got an error (lc_collate cluster values do not match:  old "C", new "en_US.UTF-8") when I tried to do the pg_upgrade step.

     

    This was an extremely helpful reply and you saved me from an otherwise difficult conversation ("What happened to all my contacts?").

     

    Thanks!

  • by igmt,

    igmt igmt Jan 23, 2015 12:43 AM in response to pwinkeler
    Level 1 (0 points)
    Jan 23, 2015 12:43 AM in response to pwinkeler

    Hi, I have the same issue and Linc Davis pointed me to this thread from Re: Server Calendar migration failed.

    Your solution already got me somewhere, so thanks for that

    I need to note that my machine is on Yosemite (10.10.1) and I'm running Server 4.

    The only issue I have is that I get an error message when running pg_upgrade, the log says:

     

    -----------------------------------------------------------------
      pg_upgrade run on Fri Jan 23 09:31:30 2015
    -----------------------------------------------------------------
    
    
    command: "/Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2/pg_ctl" -w -l "pg_upgrade_server.log" -D "/Library/Server/Calendar and Contacts/Data/Database.xpg/b$
    waiting for server to start....LOG:  database system was interrupted; last known up at 2013-11-02 12:21:52 CET
    LOG:  creating missing WAL directory "pg_xlog/archive_status"
    LOG:  starting archive recovery
    LOG:  restored log file "000000010000000000000002" from archive
    LOG:  invalid magic number D075 in log file 0, segment 2, offset 0
    LOG:  invalid checkpoint record
    FATAL:  could not locate required checkpoint record
    HINT:  If you are not restoring from a backup, try removing the file "/Library/Server/Calendar and Contacts/Data/Database.xpg/broken-cluster.pg/backup_label".
    LOG:  startup process (PID 12315) exited with exit code 1
    LOG:  aborting startup due to startup process failure
    ........................................................... stopped waiting
    pg_ctl: could not start server
    Examine the log output.
    

     

    The log talks about removing the backup_label file. When I do that the log says the following:

     

    -----------------------------------------------------------------
      pg_upgrade run on Fri Jan 23 09:41:34 2015
    -----------------------------------------------------------------
    
    
    command: "/Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2/pg_ctl" -w -l "pg_upgrade_server.log" -D "/Library/Server/Calendar and Contacts/Data/Database.xpg/b$
    waiting for server to start....LOG:  database system was interrupted; last known up at 2013-11-02 12:21:52 CET
    LOG:  starting archive recovery
    LOG:  invalid magic number D075 in log file 0, segment 2, offset 0
    LOG:  invalid primary checkpoint record
    LOG:  could not open file "pg_xlog/000000010000000000000001" (log file 0, segment 1): No such file or directory
    LOG:  invalid secondary checkpoint record
    PANIC:  could not locate a valid checkpoint record
    FATAL:  the database system is starting up
    .LOG:  startup process (PID 15068) was terminated by signal 6: Abort trap
    LOG:  aborting startup due to startup process failure
     stopped waiting
    pg_ctl: could not start server
    Examine the log output.
    
    

     

    Thanks already for the help

  • by igmt,

    igmt igmt Jan 23, 2015 1:21 AM in response to igmt
    Level 1 (0 points)
    Jan 23, 2015 1:21 AM in response to igmt

    Ok sorted this one out. Found a cluster.pg from before the previous upgrade (last week) and that did the trick.

  • by daveyostcom,

    daveyostcom daveyostcom May 19, 2015 2:18 PM in response to pwinkeler
    Level 1 (10 points)
    iPad
    May 19, 2015 2:18 PM in response to pwinkeler

    1. Quit or force quit Sever

    2. drag "/Library/Calendar and Contacts" to the trash

    3. Launch Server again.

     

    Thanks to an Apple Enterprise Support person (wonderful guy).

  • by hozra,

    hozra hozra Jul 23, 2015 5:27 AM in response to pwinkeler
    Level 1 (0 points)
    Jul 23, 2015 5:27 AM in response to pwinkeler

    Having an 10.10.4 Calendar Server stuck in "Starting" I tried Your tip. All went well until the step "Finish the upgrade..."

     

    sudo -u _calendar /Applications/Server.app/Contents/ServerRoot/usr/bin/pg_upgrade -b /Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2 -B /Applications/Server.app/Contents/ServerRoot/usr/bin -d /Library/Server/Calendar\ and\ Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade/ -D /Library/Server/Calendar\ and\ Contacts/Data/Database.xpg/cluster.pg

     

    check for "/Library/Server/Calendar and Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade" failed: No such file or directory

     

    Failure, exiting

     

    The tip from daveyostecom didn't work.

    All other services are functioning correctly, only Calendar and Contacts missing

     

    XServe 2,26 GHz Quad-Core Intel Xeon, 20 GB Ram, Yosemite 10.10.4, Server 4.1.3

  • by Jerry Spring,

    Jerry Spring Jerry Spring Aug 21, 2015 6:19 AM in response to pwinkeler
    Level 1 (14 points)
    Servers Enterprise
    Aug 21, 2015 6:19 AM in response to pwinkeler

    Hello there,

     

    I have updated my 9.2.3 Server to 10.10.5 and run into the same problems likes discussed in that thread.

    Using pwinkelers hint above leads to a problem with different SQL database versions that I can not solve.

     

    Did anyone got a answer for that problem?

     

    The log shows the problem of the different database structures.

     

    Still there is a seems a second server running. How could PostgreSQL could be stopped?

     

    -----

    command: "/Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2/pg_ctl" -w -l "pg_upgrade_server.log" -D "/Library/Server/Calendar and Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade/" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000  -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1

    pg_ctl: another server might be running; trying to start server anyway

    waiting for server to start....FATAL:  database files are incompatible with server

    DETAIL:  The data directory was initialized by PostgreSQL version 9.3, which is not compatible with this version 9.2.8.

    ........................................................... stopped waiting

    pg_ctl: could not start server

  • by test...testing123,

    test...testing123 test...testing123 Oct 1, 2015 8:27 AM in response to Jerry Spring
    Level 1 (0 points)
    Oct 1, 2015 8:27 AM in response to Jerry Spring

    Exactly the same issue as you Jerry...

     

    waiting for server to start....FATAL:  database files are incompatible with server

    DETAIL:  The data directory was initialized by PostgreSQL version 9.3, which is not compatible with this version 9.2.13.

    ........................................................... stopped waiting

  • by jenniwren,

    jenniwren jenniwren Apr 22, 2016 2:45 PM in response to pwinkeler
    Level 1 (4 points)
    Apr 22, 2016 2:45 PM in response to pwinkeler

    I was excited to see this post as my contacts server has stopped working since I upgraded my server on Sunday to Server 3.2.2.  I found this post after looking in the error log and seeing something about postgres not getting started.  I'm definitely no expert, but I kind of understand what's happening in the suggested list of actions above, so I decided it couldn't hurt to try.  To emphasize, it is CONTACTS that aren't working for me.  Perhaps Calendars are in the same boat, but I don't use them so don't know.

     

    I have already tried throwing the server app in the trash and restoring it - that did not fix anything and when doing the server setup it gave me the error message about configuration having failed.

     

    I followed the steps through to "# Finish the upgrade job that never completed..." but when I ran the pg_upgrade command I got an error:

     

    Database.xpg root# sudo -u _calendar /Applications/Server.app/Contents/ServerRoot/usr/bin/pg_upgrade -b /Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2 -B /Applications/Server.app/Contents/ServerRoot/usr/bin -d /Library/Server/Calendar\ and\ Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade/ -D /Library/Server/Calendar\ and\ Contacts/Data/Database.xpg/cluster.pg

    Performing Consistency Checks

    -----------------------------

    Checking cluster versions                                  ok

     

    *failure*

    Consult the last few lines of "pg_upgrade_server.log" for

    the probable cause of the failure.

     

    connection to database failed: could not connect to server: No such file or directory

      Is the server running locally and accepting

      connections on Unix domain socket "/Library/Server/Calendar and Contacts/Data/Database.xpg/.s.PGSQL.50432"?

     

    could not connect to old postmaster started with the command:

    "/Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2/pg_ctl" -w -l "pg_upgrade_server.log" -D "/Library/Server/Calendar and Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade/" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directory='/Library/Server/Calendar and Contacts/Data/Database.xpg'" start

    Failure, exiting

     

    Taking a look at the last few lines of "pg_upgrade_server.log" I saw this:

    -----------------------------------------------------------------

      pg_upgrade run on Fri Apr 22 14:27:33 2016

    -----------------------------------------------------------------

    command: "/Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2/pg_ctl" -w -l "pg_upgrade_server.log" -D "/Library/Server/Calendar and Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade/" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directory='/Library/Server/Calendar and Contacts/Data/Database.xpg'" start >> "pg_upgrade_server.log" 2>&1

    waiting for server to start....LOG:  database system was interrupted; last known up at 2013-11-26 15:11:55 PST

    LOG:  starting archive recovery

    cp: ../backup/000000010000000000000002: No such file or directory

    LOG:  could not open file "pg_xlog/000000010000000000000002" (log file 0, segment 2): No such file or directory

    LOG:  invalid checkpoint record

    FATAL:  could not locate required checkpoint record

    HINT:  If you are not restoring from a backup, try removing the file "/Library/Server/Calendar and Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade/backup_label".

    LOG:  startup process (PID 37871) exited with exit code 1

    LOG:  aborting startup due to startup process failure

    ........................................................... stopped waiting

    pg_ctl: could not start server

    Examine the log output.

     

    I renamed the file "/Library/Server/Calendar and Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade/backup_label" and tried running the pg_upgrade command again, but got the same error message.  This time, the last few lines of "pg_upgrade_server.log" are:

    -----------------------------------------------------------------

      pg_upgrade run on Fri Apr 22 14:14:06 2016

    -----------------------------------------------------------------

    command: "/Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2/pg_ctl" -w -l "pg_upgrade_server.log" -D "/Library/Server/Calendar and Contacts/Data/Database.xpg/cluster.pg.original_for_upgrade/" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directory='/Library/Server/Calendar and Contacts/Data/Database.xpg'" start >> "pg_upgrade_server.log" 2>&1

    waiting for server to start....LOG:  database system was interrupted; last known up at 2013-11-26 15:11:55 PST

    LOG:  starting archive recovery

    LOG:  could not open file "pg_xlog/000000010000000000000002" (log file 0, segment 2): No such file or directory

    LOG:  invalid primary checkpoint record

    LOG:  could not open file "pg_xlog/000000010000000000000002" (log file 0, segment 2): No such file or directory

    LOG:  invalid secondary checkpoint record

    PANIC:  could not locate a valid checkpoint record

    LOG:  startup process (PID 37753) was terminated by signal 6: Abort trap

    LOG:  aborting startup due to startup process failure

    ........................................................... stopped waiting

    pg_ctl: could not start server

    Examine the log output.

     

    Is there any easy solution for this?  Am I doing the right steps for resolving the issue with cardav not working as opposed to caldav?  We really need our contacts back!

     

    What do I need to do to triage having run the first few commands and not being able to finish? My guess would be that I can just work backwards, but I don't want to make anything worse!

     

    I will be so grateful for any help with this

     

    Addendum:

    Database.xpg root# ls backup

    000000010000000000000002.00000028.backup 000000010000000000000005 00000001000000000000006D

    000000010000000000000003 000000010000000000000006 base_backup

    000000010000000000000004 00000001000000000000006B

    000000010000000000000004.00000028.backup 00000001000000000000006C