Rusty Ross

Q: Server 3.2.1 (10.9.5) – Postgres won't start

After upgrading from Server 3.1.2 (Mavericks 10.9.3) to Server 3.2.1 (Mavericks 10.9.5), I cannot start Postgres:

 

# serveradmin start postgres

postgres:error = "CANNOT_START_SERVICE_TIMEOUT_ERR"

 

Here is what I see in system.log:

 

Sep 18 18:36:20 server com.apple.serverd[78]: ERROR: SMJobSubmit: The operation couldn’t be completed. (kSMErrorDomainLaunchd error 9 - The job dictionary specifies that it is disabled.)

Sep 18 18:36:20 server serveradmin[2249]: servermgr_postgres: waiting for postgres to respond

Sep 18 18:36:24 server serveradmin[2249]: servermgr_postgres: Timed out trying to start postgres.

Sep 18 18:36:24 server serveradmin[2249]: posting dist not

 

Not sure what to do from here. Postgres was running fine under 10.9.3.

 

 

Rusty

Posted on Sep 18, 2014 3:38 PM

Close

Q: Server 3.2.1 (10.9.5) – Postgres won't start

  • All replies
  • Helpful answers

Previous Page 2 of 7 last Next
  • by essandess,

    essandess essandess Sep 19, 2014 10:35 AM in response to Erik Hagemeijer
    Level 1 (28 points)
    Applications
    Sep 19, 2014 10:35 AM in response to Erik Hagemeijer

    Have a look at your logs in /Library/Logs/PostgreSQL/PostgreSQL.log

     

    What I did to troubleshoot was to pull out the actual command that serveradmin forms from the plists /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/org.po stgresql.postgres.plist and /Library/Server/PostgreSQL/Config/org.postgresql.postgres.plist, then run that command as _postgres from the directory /Library/Server/PostgreSQL/Data:

     

    su -

    cd /Library/Server/PostgreSQL/Data

    sudo -u _postgres <long command>.

     

    The fix to this problem should be a simple three-step fix of (1) using "unix_socket_directories" in /Library/Server/PostgreSQL/Config/org.postgresql.postgres.plist, (2) creating a new directory /Library/Server/PostgreSQL/Data with the correct permissions, (3) running initdb, and then using serveradmin to start postgres.

  • by essandess,

    essandess essandess Sep 19, 2014 10:40 AM in response to Erik Hagemeijer
    Level 1 (28 points)
    Applications
    Sep 19, 2014 10:40 AM in response to Erik Hagemeijer

    Also, I don't see that listen_address is specified in your serveradmin settings -- I don't know if this is a requirement. Try:

     

    sudo serveradmin set postgres:listen_address = "127.0.0.1,::1"

    sudo serveradmin start postgres

  • by Ali Kaylan,

    Ali Kaylan Ali Kaylan Sep 19, 2014 12:00 PM in response to Erik Hagemeijer
    Level 1 (5 points)
    Sep 19, 2014 12:00 PM in response to Erik Hagemeijer

    Thanks to Rusty and Erik, I managed to go through this fix. Now I get

     

    postgres:state = "RUNNING"

     

    But mail is still down despite indicating "on" in the server. For posterity, this is what I did after restoring everything:

     

    sh-3.2# chown -R _postgres:_postgres /Library/Server/PostgreSQL

    sh-3.2# cd /Library/Server/PostgreSQL

    sh-3.2# ls -l

    total 0

    drwxr-xr-x   3 _postgres  _postgres  102 Sep 19 12:44 Config

    drwx------  18 _postgres  _postgres  612 Sep 19 12:44 Data

    sh-3.2# ditto /Library/Server/PostgreSQL/Data /Library/Server/PostgreSQL/DataOld

    sh-3.2# ls -l

    total 0

    drwxr-xr-x   3 _postgres  _postgres  102 Sep 19 12:44 Config

    drwx------  18 _postgres  _postgres  612 Sep 19 12:44 Data

    drwx------  18 _postgres  _postgres  612 Sep 19 12:44 DataOld

    sh-3.2# rm -rf /Library/Server/PostgreSQL/Data/*

    sh-3.2# sudo -u _postgres /Applications/Server.app/Contents/ServerRoot/usr/bin/initdb /Library/Server/PostgreSQL/Data -E utf8 --lc-collate=C --lc-ctype=C

    The files belonging to this database system will be owned by user "_postgres".

    This user must also own the server process.

     

    The database cluster will be initialized with locales

      COLLATE:  C

      CTYPE:    C

      MESSAGES: en_US.UTF-8

      MONETARY: en_US.UTF-8

      NUMERIC:  en_US.UTF-8

      TIME:     en_US.UTF-8

    The default text search configuration will be set to "english".

     

    Data page checksums are disabled.

     

    fixing permissions on existing directory /Library/Server/PostgreSQL/Data ... ok

    creating subdirectories ... ok

    selecting default max_connections ... 100

    selecting default shared_buffers ... 128MB

    creating configuration files ... ok

    creating template1 database in /Library/Server/PostgreSQL/Data/base/1 ... ok

    initializing pg_authid ... ok

    initializing dependencies ... ok

    creating system views ... ok

    loading system objects' descriptions ... ok

    creating collations ... ok

    creating conversions ... ok

    creating dictionaries ... ok

    setting privileges on built-in objects ... ok

    creating information schema ... ok

    loading PL/pgSQL server-side language ... ok

    vacuuming database template1 ... ok

    copying template1 to template0 ... ok

    copying template1 to postgres ... ok

    syncing data to disk ... ok

     

    WARNING: enabling "trust" authentication for local connections

    You can change this by editing pg_hba.conf or using the option -A, or

    --auth-local and --auth-host, the next time you run initdb.

     

    Success. Since you appear to be using the default database,

    the system-wide instance of postgres will be launched on-demand by

    the various services which use it.

     

    If necessary, you can now start the database server using:

     

        serveradmin start postgres

     

    See webappctl(8) to learn how to launch postgres on-demand for

    your web application.

    sh-3.2#  mv /Volumes/TinTin/Applications/Server.app/Contents/ServerRoot/usr/bin/postgres /Volumes/TinTin/Applications/Server.app/Contents/ServerRoot/usr/bin/postgres_of f

    sh-3.2# mv /Volumes/TinTin/Applications/Server.app/Contents/ServerRoot/usr/bin/postgres_re al /Volumes/TinTin/Applications/Server.app/Contents/ServerRoot/usr/bin/postgres

    sh-3.2# sudo -u _postgres /Applications/Server.app/Contents/ServerRoot/usr/bin/pg_upgrade -b /Volumes/TinTin/Applications/Server.app/Contents/ServerRoot/usr/bin -B /Applications/Server.app/Contents/ServerRoot/usr/bin -d /Library/Server/PostgreSQL/DataOld -D /Library/Server/PostgreSQL/Data

    Performing Consistency Checks

    -----------------------------

    Checking cluster versions                                   ok

    Checking database user is a superuser                       ok

    Checking for prepared transactions                          ok

    Checking for reg* system OID user data types                ok

    Checking for contrib/isn with bigint-passing mismatch       ok

    Creating dump of global objects                             ok

    Creating dump of database schemas

                                                                ok

    Checking for presence of required libraries                 ok

    Checking database user is a superuser                       ok

    Checking for prepared transactions                          ok

     

    If pg_upgrade fails after this point, you must re-initdb the

    new cluster before continuing.

     

    Performing Upgrade

    ------------------

    Analyzing all rows in the new cluster                       ok

    Freezing all rows on the new cluster                        ok

    Deleting files from new pg_clog                             ok

    Copying old pg_clog to new server                           ok

    Setting next transaction ID for new cluster                 ok

    Setting oldest multixact ID on new cluster                  ok

    Resetting WAL archives                                      ok

    Setting frozenxid counters in new cluster                   ok

    Restoring global objects in the new cluster                 ok

    Adding support functions to new cluster                     ok

    Restoring database schemas in the new cluster

                                                                ok

    Removing support functions from new cluster                 ok

    Copying user relation files

                                                                ok

    Setting next OID for new cluster                            ok

    Sync data directory to disk                                 ok

    Creating script to analyze new cluster                      ok

    Creating script to delete old cluster                       ok

     

    Upgrade Complete

    ----------------

    Optimizer statistics are not transferred by pg_upgrade so,

    once you start the new server, consider running:

        analyze_new_cluster.sh

     

    Running this script will delete the old cluster's data files:

        delete_old_cluster.sh

    sh-3.2# mv /Volumes/TinTin/Applications/Server.app/Contents/ServerRoot/usr/bin/postgres /Volumes/Tintin/Applications/Server.app/Contents/ServerRoot/usr/bin/postgres_re al

    sh-3.2# mv /Volumes/TinTin/Applications/Server.app/Contents/ServerRoot/usr/bin/postgres_of f /Volumes/TinTin/Applications/Server.app/Contents/ServerRoot/usr/bin/postgres

    sh-3.2#

  • by TonyPHX_623,

    TonyPHX_623 TonyPHX_623 Sep 19, 2014 2:44 PM in response to Ali Kaylan
    Level 1 (5 points)
    Sep 19, 2014 2:44 PM in response to Ali Kaylan

    All,


    It gets even more interesting if you simply install a new Mavericks host with Server from scratch.  postgreSQL is essentially missing and there are various instances of it in each "service" that needs it.  They fundamentally changed where it is installed for net new installations.  The migration is hosed for sure, but for a real eye opener, do a fresh install.

  • by mylnir,

    mylnir mylnir Sep 19, 2014 5:42 PM in response to Rusty Ross
    Level 1 (0 points)
    Sep 19, 2014 5:42 PM in response to Rusty Ross

    I followed the postgres reinitialization steps, though unfortunately, I didn't have a handy DB backup, so a few steps didn't apply to me.  The other gotcha I had though, was my calendar/contacts directory was completely corrupted and unable to start or stop, even surviving reboots.

     

    Moving the "/Library/Server/Calendar and Contacts" directory to the side was required to clear the problem.  Now the joy of reinstalling data.

     

    Repairing postgres may not be enough.

  • by Dfundy,

    Dfundy Dfundy Sep 19, 2014 10:22 PM in response to mylnir
    Level 1 (15 points)
    Sep 19, 2014 10:22 PM in response to mylnir

    Trying to follow the steps suggested by Rusty and Erik, I hit the wall with attempt to use sudo -u _postgress  .....

    command. The system outputs:

     

    sudo: unable to change to runas uid (216, 216): Operation not permitted

     

    Any suggestions what might be the problem?

     

    Many thanks

     

    Daniel

  • by John Caradimas,

    John Caradimas John Caradimas Sep 20, 2014 1:05 AM in response to Rusty Ross
    Level 1 (19 points)
    Servers Enterprise
    Sep 20, 2014 1:05 AM in response to Rusty Ross

    Strange, I just upgraded (maybe downgraded?) my server to the latest versions of Maverick and Server.app and postgres seems to be working fine.

     

    sh-3.2# serveradmin status postgres

    postgres:state = "RUNNING"

     

    Is there something specific I should check to make sure?

  • by vwbeta,

    vwbeta vwbeta Sep 20, 2014 3:21 AM in response to Erik Hagemeijer
    Level 1 (0 points)
    Sep 20, 2014 3:21 AM in response to Erik Hagemeijer

    just to confirm. i have the same issue after the upgrade to 3.2.1

  • by TonyPHX_623,

    TonyPHX_623 TonyPHX_623 Sep 20, 2014 6:36 AM in response to John Caradimas
    Level 1 (5 points)
    Sep 20, 2014 6:36 AM in response to John Caradimas

    Running and responding are two different things.

  • by Rusty Ross,

    Rusty Ross Rusty Ross Sep 20, 2014 7:13 AM in response to Dfundy
    Level 2 (175 points)
    Sep 20, 2014 7:13 AM in response to Dfundy

    DFundy:  I would become root (sudo su) first and run all of this process from there. Also make sure you spell the postgres user correctly (_postgres not _postgress).

     

    John Caradimas:  Run "psql" and I think you will likely find that postgres is not responding, even if serveradmin reports the status as "RUNNING".

  • by John Caradimas,

    John Caradimas John Caradimas Sep 20, 2014 10:00 AM in response to Rusty Ross
    Level 1 (19 points)
    Servers Enterprise
    Sep 20, 2014 10:00 AM in response to Rusty Ross

    Rusty Ross, thanks for your suggestion. I tried psql and here is what I got:

     

    psql: could not connect to server: No such file or directory

        Is the server running locally and accepting

        connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?

     

    So I assume I suffer from the described problem.

     

    One more stupid question, if you do not  mind. Is there anyway to tell if postgres is used in my system? I mean I haven't seen the system misbehaving in any way with postgres not working correctly, so shall I assume that I am not using it anywhere?

     

    Again, thanks.

  • by Rusty Ross,

    Rusty Ross Rusty Ross Sep 20, 2014 10:07 AM in response to John Caradimas
    Level 2 (175 points)
    Sep 20, 2014 10:07 AM in response to John Caradimas

    Many OS X Server processes (Wiki, Profile Manager, etc) use postgres, but that is actually separate installation and I don't believe it is problematic in this version. (On the servers I have tested, it has not been.)

     

    The postgres service being discussed in this thread (the one typically started via "serveradmin start postgres", is typically for custom and/or user installed software that needs postgres as a database backend. I would venture to guess that if you haven't intentionally built something on your server which uses postgres, then you probably aren't affected by the issue being discussed here. Your mileage may vary, of course.

  • by essandess,

    essandess essandess Sep 20, 2014 10:08 AM in response to John Caradimas
    Level 1 (28 points)
    Applications
    Sep 20, 2014 10:08 AM in response to John Caradimas

    The command that the (plist corrected) serveradmin runs is:

     

    sudo -u _postgres /Applications/Server.app/Contents/ServerRoot/usr/bin/postgres -D /Library/Server/PostgreSQL/Data -c listen_addresses=127.0.0.1,::1 -c log_connections=on -c log_directory=/Library/Logs/PostgreSQL -c log_filename=PostgreSQL.log -c log_line_prefix=%t  -c log_lock_waits=on -c log_statement=ddl -c logging_collector=on -c unix_socket_directories=/private/var/pgsql_socket -c unix_socket_group=_postgres -c unix_socket_permissions=0770

     

    cd to /Library/Server/PostgreSQL/Data as root and run this command. If it works, then the serveradmin start postgres command should work. If it doesn't, look at the log file to see why postgres isn't starting.

  • by Victor Pang,

    Victor Pang Victor Pang Sep 20, 2014 12:39 PM in response to Rusty Ross
    Level 1 (110 points)
    Sep 20, 2014 12:39 PM in response to Rusty Ross

    The old PostgreSQL 9.2 binaries are located at "/Applications/Server.app/Contents/ServerRoot/usr/libexec/postgresql9.2/" for OS X Server 3.2.1.

  • by Rusty Ross,

    Rusty Ross Rusty Ross Sep 20, 2014 12:57 PM in response to Victor Pang
    Level 2 (175 points)
    Sep 20, 2014 12:57 PM in response to Victor Pang

    Nice find. I'll use that path to factor a more generic upgrade script and post it here later.

Previous Page 2 of 7 last Next