migrate wiki from MLS to MAVS
Dear Folks,
I want to provide a solution on how to migrate the Wiki from a Mountain Lion Server to a Mavericks Server. Both Servers are running but in different locations and they can't communicate directly with each other. Upgrading the MLS was no option - after what I had read in hundreds of threads. The MLS is a 10.8.5 (with the latest version of server.app up until the 15th of October 2013) and Mavericks a 10.9.1 (with the latest version of server.app up until New Years Eve).
I hadn't done it without the help of Andreas from Metalab in Vienna (lovely Hackspace) - so the grace goes to him!!
First things first - he isolated 2 issues on MLS - revolving postgresql Version AND Socket! Thank you Apple-developers you did a great job here 😢 - since this product is called „The Server for everyone“ I believe either the title is incorrect or some guys haven’t understood what the a server for everyone implies - since everyone isn’t a geek, nerd, developer etc. - think about it!
Trying to connect to psql on MLS:
bash-3.2# psql
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?
What?!
bash-3.2# serveradmin status postgres
postgres:state = "RUNNING"
What?!
So he started digging on MLS:
bash-3.2# netstat -na|grep LISTEN
nothing tangible here - nothing ist listening to 5432
Dig deeper on MLS:
bash-3.2# lsof | grep postgre
#output omitted for clarity
postgres_ 2546 _postgres 10u unix 0xa319efaa0af41357 0t0 /Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432
postgres_ 2547 _postgres 10u unix 0xa319efaa0af3fbe7 0t0 /Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432
#output omitted for clarity
postgres_ 2548 _postgres 10u unix 0xa319efaa0af3fa57 0t0 /Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432
#output omitted for clarity
postgres_ 2549 _postgres 10u unix 0xa319efaa0af3f8c7 0t0 /Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432
#output omitted for clarity
postgres_ 3102 _postgres 10u unix 0xa319efaa0b220357 0t0 /Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432
#output omitted for clarity
Hold the horses still - that got him thinking! How does that fit to the output of the psql-command just seconds earlier (and I literally mean seconds!) Can you see it?
The truth is, the Socket is at:
/Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432
but is expected by psql to be in /var/psql_socket/.s.PGSQL.5432!
Expected What?!
Verified it again on MLS:
bash-3.2# psql
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?
bash-3.2# psql --help
Yep - when postgresql was compiled obviously somebody was sitting on his fingers - during implementation nobody had told psql the path for the socket at the "new" location?! Ergo - psql is looking for the socket in the default location - where it is not.
Then, with the help, he managed it to get in.
Trying to get in on MLS:
bash-3.2# psql -p "/Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432"
psql: invalid port number: "/Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432"
Trying again on MLS:
bash-3.2# psql -h "/Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432"
psql: could not connect to server: Not a directory
Is the server running locally and accepting
connections on Unix domain socket "/Library/Server/PostgreSQL For Server Services/Socket/.s.PGSQL.5432/.s.PGSQL.5432"?
And again - almost there on MLS:
bash-3.2# psql -h "/Library/Server/PostgreSQL For Server Services/Socket/" -p 5432
psql: FATAL: role "root" does not exist
Typo:
bash-3.2# psql -h "/Library/Server/PostgreSQL For Server Services/Socket/" -p 5432 -u colab
psql: invalid option -- u
Try "psql --help" for more information.
And - tata - he was in on MLS:
bash-3.2# psql -h "/Library/Server/PostgreSQL For Server Services/Socket/" -p 5432 template1 collab
psql (9.1.9, server 9.2.4)
WARNING: psql version 9.1, server version 9.2.
Some psql features might not work.
Type "help" for help.
template1=#
Lets find the databases and roles on MLS:
template1=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------------+------------+----------+---------+-------+------------------ -------
caldav | caldav | UTF8 | C | C |
collab | collab | UTF8 | C | C |
device_management | _devicemgr | UTF8 | C | C |
postgres | _postgres | UTF8 | C | C |
template0 | _postgres | UTF8 | C | C | =c/_postgres +
| | | | | _postgres=CTc/_postgres
template1 | _postgres | UTF8 | C | C | _postgres=CTc/_postgres+
| | | | | =c/_postgres
webauth | webauth | UTF8 | C | C |
(7 rows)
And there are the roles on MLS:
template1=# \du
List of roles
Role name | Attributes | Member of
------------+------------------------------------------------+-----------
_devicemgr | Create DB | {}
_postgres | Superuser, Create role, Create DB, Replication | {}
caldav | Create DB | {}
collab | Superuser, Create role, Create DB | {}
webauth | Superuser, Create role, Create DB | {}
Aaaaah - so there is a role collab and a database collab! Lets connect to the "right" database and dive into it on MLS.
template1=# \q
bash-3.2# psql -h "/Library/Server/PostgreSQL For Server Services/Socket/" -p 5432 collab collab
psql (9.1.9, server 9.2.4)
WARNING: psql version 9.1, server version 9.2.
Some psql features might not work.
Type "help" for help.
collab=# \d
List of relations
Schema | Name | Type | Owner
--------+-----------------------------+-------+--------
public | blog_entity | table | collab
public | document_entity | table | collab
public | entity | table | collab
public | entity_acls | table | collab
public | entity_acls_defaults | table | collab
public | entity_attrs | table | collab
public | entity_changesets | table | collab
public | entity_comment | table | collab
public | entity_lock | table | collab
public | entity_preview | table | collab
public | entity_private_attrs | table | collab
public | entity_tag | table | collab
public | entity_type | table | collab
public | file_entity | table | collab
public | filedata_entity | table | collab
public | filename_reservation | table | collab
public | global_settings | table | collab
public | groups | table | collab
public | migration_entity | table | collab
public | migration_status | table | collab
public | migrationplaceholder_entity | table | collab
public | notification | table | collab
public | page_entity | table | collab
public | podcast_entity | table | collab
public | podcast_episode_entity | table | collab
public | preview_queue | table | collab
public | project_entity | table | collab
public | relationship | table | collab
public | savedquery_entity | table | collab
public | search_index | table | collab
public | search_stat | table | collab
public | session | table | collab
public | subscription | table | collab
public | user_activity | table | collab
public | user_entity | table | collab
public | user_entity_favorites | table | collab
public | user_entity_read_status | table | collab
public | user_entity_unread_status | table | collab
public | user_entity_updates | table | collab
public | user_entity_watched | table | collab
public | user_readall_time | table | collab
(41 rows)
That looks fantastic! Lets start dumping with some help through pg_dump --help and then issuing the following on MLS:
bash-3.2# pg_dump -h "/Library/Server/PostgreSQL For Server Services/Socket/" -p 5432 -f /Volumes/USBSTICK/wikidatabase -U collab collab
pg_dump: server version: 9.2.4; pg_dump version: 9.1.9
pg_dump: aborting because of server version mismatch
What?! How could that be?! The commands refer to an older version?!
Start digging again on MLS!
bash-3.2# which psql
/usr/bin/psql
bash-3.2# which pg_dump
/usr/bin/pg_dump
That's where I stepped in and told Andreas "Hey - ahm - I know of a path completely different…. /Applications/Server.app/.….". Thanks mate!
Dig again on MLS!
bash-3.2# /Applications/Server.app/Contents/ServerRoot/usr/bin/pg_dump --version
pg_dump (PostgreSQL) 9.2.4
Lovely - that one looks good - lets use it. All of that got us finally to the right command on MLS.
bash-3.2# /Applications/Server.app/Contents/ServerRoot/usr/bin/pg_dump -h "/Library/Server/PostgreSQL For Server Services/Socket/" -p 5432 -f /Volumes/USBSTICK/wikidatabase -U collab collab
There it is - the whole Database on the stick - finally - it took Andreas only 5 Minutes - approximately? I am just baffled.
Then I copied the Folders from the /Library/Server/Wiki/FileData of the MLS to the exact same location on the MAVS and sat permission - navigating to /Library/Server/Wiki first:
bash-3.2# chown -R _teamsserver:_teamsserver FileData/
bash-3.2# chmod -R +a "_www allow read" FileData/
I compared the the collab DB on the MLS with the MAVS collab DB to see the differences - BTW - the Socket has moved again on the MLS! Move up in this documentary and you’ll see on how to find it on MAVS.
bash-3.2# lsof | grep 5432
#output omitted for clarity
postgres_ 382 _teamsserver 3u unix 0x399cdd6eeea1efe1 0t0 /Library/Server/Wiki/PostgresSocket/.s.PGSQL.5432
#output omitted for clarity
bash-3.2# psql -h "/Library/Server/Wiki/PostgresSocket/" -p 5432 collab collab
psql (9.2.4)
Type "help" for help.
collab=# \d
List of relations
Schema | Name | Type | Owner
--------+-----------------------------+-------+--------
public | adc_device_entity | table | collab
public | adc_team_entity | table | collab
public | blog_entity | table | collab
public | bot_entity | table | collab
public | botgroup_entity | table | collab
public | botrun_entity | table | collab
public | build_agent_registry | table | collab
public | document_entity | table | collab
public | email_notification | table | collab
public | entity | table | collab
public | entity_acls | table | collab
public | entity_acls_defaults | table | collab
public | entity_attrs | table | collab
public | entity_auditlog | table | collab
public | entity_changesets | table | collab
public | entity_comment | table | collab
public | entity_lock | table | collab
public | entity_preview | table | collab
public | entity_private_attrs | table | collab
public | entity_tag | table | collab
public | entity_type | table | collab
public | file_entity | table | collab
public | filedata_entity | table | collab
public | filename_reservation | table | collab
public | global_settings | table | collab
public | groups | table | collab
public | migration_entity | table | collab
public | migration_status | table | collab
public | migrationplaceholder_entity | table | collab
public | page_entity | table | collab
public | podcast_entity | table | collab
public | podcast_episode_entity | table | collab
public | preview_queue | table | collab
public | relationship | table | collab
public | savedquery_entity | table | collab
public | scm_commit_entity | table | collab
public | scm_server | table | collab
public | scmrepogroup_entity | table | collab
public | search_index | table | collab
public | search_stat | table | collab
public | session | table | collab
public | subscription | table | collab
public | timeseries | table | collab
public | timeseries_toc | table | collab
public | user_activity | table | collab
public | user_entity | table | collab
public | user_entity_favorites | table | collab
public | user_entity_read_status | table | collab
public | user_entity_unread_status | table | collab
public | user_entity_updates | table | collab
public | user_entity_watched | table | collab
public | user_readall_time | table | collab
public | visible_entity_tag | view | collab
public | wiki_entity | table | collab
public | work_queue | table | collab
public | work_schedule | table | collab
public | work_schedule_recurrence | table | collab
public | work_schedule_status | table | collab
(58 rows)
Hmmm - 41 tables in the old, and 58 in the new. What am I going to do? To be able to roll back I dumped the MAVS DB before I would do anything else.
I had to decide between trying to take the old MLS DB and just pg_restore it into the MAVS DB, or - trying to upgrade the MLS instance to MAVS. I was afraid doing the upgrade since the MLS was a mess. At the end I would decide to go for the latter - since I had dumps. It took a while but - hey - it worked. Everything in place! Wiki running - everything else, too! The database has now 58 tables - ok - lets dump it. Notice again - no need to specify the proper version of pg_dump, but the Socket has changed from MLS to MAVS - again - you could see this when I connected with psql as well!
Please use (-F c) compression when dumping - otherwise you’ll receive a nasty error when importing it at the MAVS.
bash-3.2# pg_dump -h "/Library/Server/Wiki/PostgresSocket/" -p 5432 -F c -f /Volumes/USBSTICK/MLSafterupgrade2MAVS_collab_db_compressed.pgdump -U collab collab
I then copied the dump to the server - the Desktop of the administrativ account - I did it through screensharing which is accessible after establishing a VPN to the location the server resides in.
Unfortunately the command for pg_restore has a different syntax then the pg_dump. Use pg_dump --help to see the details. I’ll explain it quickly:
- -c is to clean entries in the target tables (you must not use capital c)
- -d specifies the database into which you would like to restore
- -h specifies (again) where the DB is - the socket
- -p the port to use
- -U is the role that likes to work on the DB
- you must not use -f since it’s not needed
bash-3.2# pg_restore -c -d collab -h "/Library/Server/Wiki/PostgresSocket/" -p 5432 /Volumes/OSXDATA/Users/macminiadmin/Desktop/MLScollabdb_compressed.pgdump -U collab
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 2466; 2605 16639 CAST CAST (text[] AS public.hstore)
pg_restore: [archiver (db)] could not execute query: ERROR: cannot drop cast from text[] to public.hstore because extension hstore requires it
HINT: You can drop extension hstore instead.
Command was: DROP CAST (text[] AS public.hstore);
pg_restore: [archiver (db)] Error from TOC entry 1816; 2616 16636 OPERATOR CLASS hash_hstore_ops collab
pg_restore: [archiver (db)] could not execute query: ERROR: cannot drop operator class hash_hstore_ops for access method hash because extension hstore requires it
HINT: You can drop extension hstore instead.
Command was: DROP OPERATOR CLASS public.hash_hstore_ops USING hash;
#output omitted for clarity
I saw quite a lot errors (some revolving functions etc.) - however, it worked. Wikis are present again on the MAVS. Not without the initial help of Andreas - thank you once again!
I would like to recommend to everyone joining hackerspaces in their location. I do not know them in other areas - I can only speak for the metalab in Vienna. Fantastic place with fantastic, very knowledgable people. I would also recommend reading a book like PostgreSQL 9 Admin Cookbook - this helped me quit a lot as well. It’s available through various stores - virtual or real - choose what you prefer.
Mac mini, OS X Mavericks (10.9.1)