reconstruct/rebuild

I have now spent over a day trying to figure out what the above mean and how to apply to my problem. Here's what I've found out so far.

Despite what I read, reconstruct CANNOT be run as cyrus (i.e. cyrusimap), it HAS to be run as root yet permissions of files it creates are correct.

The command I think I need to execute is 'reconstruct -m', but this is "not yet implemented".

Apple recommend 'reconstruct -i', yet omit to mention this option and what it does in the man page. In fact it creates mailboxes to match what's in mailboxes.db which is the opposite of what I need.

Running 'Rebuild" on the database in Server Admin does nothing. OK, the mailboxes.db file seems to be new, but it's always the same as the previous one, which is incorrect, hence my need to re-create it.

I am at the end of my tether here. I have all the correct mailboxes and index files for each user, but I cannot see how to rebuild the actual list of mailboxes so that clients can actually see them.

Can anyone please explain how I can recreate mailboxes.db from the mailbox file heirarchy in each user's mailstore? If I delete it before running reconstruct, I get a new file, but it's basically empty.

I've read in docs about the mailboxes file from which mailboxes.db is created, but this doesn't exist on my Leopard server. I assume it must build the .db file from a raw text file, but which one and where is it? If I could manually edit that file and rebuild the .db then I might be in with a chance of getting it working.

So, I could really do with some help here. How can I re-create a CORRECT mailboxes.db file that accurately lists the mailboxes and folders for each user? Hope someone can give me a clue here before I do something I might regret later.

MacPro - Dual 3GHz Quad Core Xeon, Mac OS X (10.4.10), 30" HD display

Posted on Dec 15, 2007 6:41 AM

Reply
10 replies

Dec 15, 2007 9:33 AM in response to pterobyte

Thanks for the reply.

I have downloaded mailbfr and will have a good look. But a cursory glance indicates that it rebuilds mailbox indexes and that is not what I need as I believe they are all correct. My problem is that the mailboxes.db file is not correct. It still lists mailboxes in locations where they no longer exist. If I run 'reconstruct -i', it recreates all those mailboxes again (but with nothing in them).

I really want to start from scratch and somehow build a new mailboxes.db as an accurate list of the mailboxes that really do exist (and contain all the mail), such that the client can them see them when connecting to the server. It seems to me that this is what would be required when recovering a saved mail store and then needing to tell the server to USE THAT and forget about everything else.

Can mailbfr build a new mailboxes.db? How is a Leopard Server admin supposed to do this without resorting to third party products?

Dec 15, 2007 11:07 AM in response to UKenGB

1. Copy all messages to a safe place. Messages are the numbered files with a dot at the end.
2. Download and use SirAdmin to delete the mailboxes. Either all or just the ones you don't want anymore. After deleting the ones you don't want anymore, make sure you also delete the user from WGM
3. Log in with an IMAP client to recreate the mailbox(es) (in case you deleted all of them)
4. Copy the messages back into the mailbox(es)
5. Run mailbfr -f

Yes, mailbfr does rebuild mailboxes.db based on the users in WGM (and if you deleted users in WGM without deleting their mailboxes with SirAdmin first those will be added as well).

SirAdmin can be found here: http://explanatorygap.net/2006/05/13/siradmin-a-gui-replacement-for-cyradm/

Dec 15, 2007 11:31 AM in response to pterobyte

Can someone perhaps explain what the following is in mailbfr.

-f | --fixdb
Fix the mail database.

-m | --fixmailbox username
Fix a single mailbox/user.

I assume the latter is to rebuild individual mailbox indexes whereas the former seems to imply mailboxes.db will be rebuilt. However, having run it, all the mailboxes have been edited, but mailboxes.db is UNCHANGED:-(

In any case all the mailboxes are now visible in the client, but the Received date on all messages is the same so I have no idea when any message was received. This is a complete mess. All I wanted was to build a new mailboxes.db? I can't understand why this seems so impossible to achieve. What is mailboxes.db built from? How come mailbfr didn't touch it? I need to understand so that when I start all over again I know how to avoid Cyrus screwing me up again.

Dec 15, 2007 11:39 AM in response to UKenGB

If you used mailbfr -f then mailboxes.db was rebuilt (together with the whole configuration directory). Before rebuilding a time-stamped copy of your former configuration folder was made. Assuming you are using default locations, you will find the time-stamped copy in /var

mailbfr -m will not rebuild the mailboxes.db

Set your mail client to display the sent date and sort by sent date if you lost the received dates.

How to add/remove mailboxes is explained in my previous post.

Dec 15, 2007 1:21 PM in response to pterobyte

Since I was actually In the var/imap directory at the time it was not immediately apparent that it had been renamed and a new imap directory created with the new files which I can now see are ideed new.

However, what exactly does mailbfr -f do? Does it ONLY re-build mailboxes.db or does it also rebuild the mailbox indices (like mailbfr -m)? I'm copying all the mail over again since the received dates are all hosed in the rebuilt mailboxes and I don't want to have to do that again. Can I rebuild mailboxes.db without losing received date? Isn't there some way to be able to rebuild mailboxes and keep stuff like received date, it's in the actual message is it not, so it's not rocket science to ensure that rebuilding mailbox info doesn't trash important info like that? I don't know about anyone else, but that seems pretty crucial to me. I take your point about using sent date, but it's not the same and that's the point.

Dec 16, 2007 12:46 AM in response to UKenGB

mailbfr is among other things a wrapper around the cyrus admin utilities. It can't do anything the cyrus tools can't, but it makes sure procedures are carried out in the right order and in a more user friendly manner. It's a shell script, so you can open it in a text editor and see for yourself.

mailbfr -f does essentially run "sudo -u "cyrususer" /usr/bin/cyrus/bin/reconstruct -i". It does reconstruct the indices, but it will first clean up and also fix permissions when done. Furthermore it detects the system it runs on and adapts parameters as needed. (As you noticed, simply running reconstruct doesn't solve your issues).

mailbfr -m does run "sudo -u "cyrususer" /usr/bin/cyrus/bin/reconstruct -r -f user/username". This however only re-indexes a single user's messages. If anything is wrong with the database it will not be able to fix it. It also makes sure permissions are set correctly.

None of above operations will modify your messages' dates, only the read/unread flags are reset by mailbfr -f. If your message dates at the file system level are incorrect, it is because messages where copied incorrectly (probably through the Finder). Use cp -p to preserve attributes.

As an aside: If you are unhappy with Apple's or Cyrus' implementation, file a bug report with them. I am not affiliated with them and am just trying to help. 😉

Dec 16, 2007 4:33 AM in response to pterobyte

I understand your position and am very grateful for the help. It's just that setting up Leopard Server has been a nightmare every step of the way, when I chose to use it as a simpler (i.e. quicker) alternative to Linux (and I wanted to keep everything Mac) It would have been MUCH quicker for me to stick with Linux.

Having got the server running I needed to move all the mail from the old Linux server (mbox format) to the new Leopard Server (Cyrus, as we know) and that has taken DAYS. After first copying the messages I found that the account with the largest amount of mail (hundreds of folders in a heirarchical sructure) had somehow been 'edited' by the system (server? client?) to change the IMAP path prefix, so all the mail was in the wrong place. This was done after the prefix was correctly set and it then occurred with another account. I am not imagining this. To avoid having to repeat the whole process I moved the mail to where it should be and then needed to rebuild JUST mailboxes.db so that the mail could be seen in its new location by the client (OSX's Mail).

First of all, it seemed incredible to me that the system could not automatically detect the mail in the new location. On the Linux servers, this was such a simple process. Move the mail then change the IMAP path prefix in the client and bingo, there it all was. Cyrus is SERIOUSLY deficient in this regard. It then transpired that the only way to rebuild mailboxes.db involved rebuilding the mailbox indexes and not only is ALL mail then 'Unread', it has lost the 'Received' date and this is an unacceptable situation when thousands of messages are stored in that very precise order. So I had to start all over again and move all the messages again. This has taken DAYS and I can assure you I am far from happy with Apple at this moment in time. The product is far from finished with other serious issues and the documentation HOPELESS. It simply paraphrases the bleeding obvious or what is stated on the screen. USELESS.

It is finally running now and I thank you for your help - I am mad at Apple (Someone needs shooting), not yourself.

Finally, where is the 'Received' date taken from? It is not taken from the filesystem, so however I copied the mail made no difference. It is the effect of reconstructing the mailboxes that causes it, but is it not possible for the date to be retained, or simply extracted from a message header so it is not lost? Recovering from a backup should re-create everything exactly as it was and the 'Received' date of email is an indispensable piece of information, possibly more so than for any other type of data. Now if mailbfr could do that....:-)

Mar 3, 2008 9:50 AM in response to UKenGB

UKenGB:

I too have embarked on moving from Linux to Mac OS X Server to make my life simpler, only to find days (no, almost weeks) wasted with mail woes.

If you want the gory details:
On RH 7.3, I was running uw-imap for years (maybe 9+ yrs). When I updated to Leopard client, I could no longer move mail from one imap server to another by dragging from my inbox to the folders (the inbox being hosted on another RH7.3 machine (this time running courier-imap, the other imap server being a RH7.3 running uw-imap). If you want to know why I had 2 different imap servers - thank Apple for that. Each subsequent version of Mail seemed to break something with my imap server(s). Therefore, I updated to courier (which uses Maildir mailbox format instead of the ol' reliable (and slower) mbox). Anyway, along comes Leopard which I happily upgrade my clients to. Turns out, Leopard Mail pops up an "Append error" when I try to move messages from inbox to another imap folder. The mail does get copied (actually not always) but the popup alert always comes - very annoying. Anyway, I decided I had to give OS X Server a try anyway to alleviate all these imap mail issues. Thus I embarked on this multi-week wasted time experience....I managed to convert the old mbox format imap mailboxes to Maildir as a precursor to converting to Cyrus on OS X Server. Next I tarred up the Maildir folder and copied to OS X Server, then shut down mail services, then untared in the location /var/spool/imap/user/ *** as docs specified. Then tried running scripts to convert from Maildir to cyrus...they "almost" worked...

Now I have all the mailboxes with valid contents, however no cyrus.index or cyrus.cache files. The cyrus.header files do exist and match the mailboxes.db for that user. I have not been able to find a tool that will recreate the index and cache files now. I believe reconstruct has told me "invalid format" --- what next??

Apr 16, 2008 4:35 AM in response to UKenGB

A method to recreate all the cyrus files can be found at http://docs.info.apple.com/article.html?artnum=107996

After creating the email in /var/spool/imap/user using a script derived from http://samage.net/dev/ executing the script below at root resulted in a working email system.

#!/bin/bash
echo "you are about to recreate all the cyrus databases. "
echo " "
echo -n "OK to proceed (y/n) ? "
read ANSWER
#echo ANSWER=$ANSWER
if [ "$ANSWER" != 'y' ]; then
echo process aborted
exit 1;
fi
echo backup the /var/imap/ ...
rm -rf /var/imap.old/
mv /var/imap /var/imap.old

echo recreate /var/imap/ ...
mkdir /var/imap
/usr/bin/cyrus/tools/mkimap
chown -R _cyrus:mail /var/imap
/usr/bin/cyrus/bin/reconstruct -i

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

reconstruct/rebuild

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.