Mail indexing between two machines

Is Mail used in more than one of these machines between synchronizations?

Note that, while rsync (or any other synchronization utility) could in theory take care of synchronizing the messages themselves, you are calling for trouble if other files that Mail uses to keep track of everything (e.g. the Envelope Index file) get modified in two or more different machines between synchronizations.

Worse yet, rsync (or any other synchronization utility) cannot really take care of synchronizing the messages themselves. If two machines download messages at different times between synchronizations, chances are that different messages get assigned the same sequence number and, hence, the same name in the filesystem. Thus, when rsync sees these files, it will think they are different versions of the same file and one of the messages will be overwritten by the other (i.e. it'll be lost).

So my advice is don't do that, unless you can ensure that Mail is allowed to run on one machine only between synchronizations.

iMac G5 17" (iSight) Mac OS X (10.4.4)

no Mail is not used while doing rsync; always careful
to quit Mail first

This is not what I was asking. My question was whether Mail was allowed to run in more than one machine between synchronizations.

let me repeat: the real issue is that this has worked
for me for years and it did NOT stop with the move to
10.4 but with some update installe over the last
months.

I strongly believe this is pure coincidence. The setup you've described is calling for trouble, and is doomed to cause precisely the kind of problems you're experiencing, or even worse, sooner or later.

If you can really ensure that Mail runs only on one machine between synchronizations, it should work. So let's suppose that you've been doing it right all this time -- note that you need to get this wrong only once to start wreaking havoc. If I understood correctly, after using Mail on one machine, but before switching to another, you quit all applications on the first, make a backup copy of it, and then use the backup to update the contents on the second before start using it, right?

Have all machines always had correct clock settings? Which version of rsync are you using? Could you provide an example of the rsync commands used, including all the options specified?

how would you recommend doing it?
(note: .mac caused all kinds of other problems which
is why i never pursued that otion)

If all you want synchronized is your mail, I would recommend using an IMAP account and let Mail and the mail server do the synchronization work for you. But it looks like you stay on one machine at a time and just want a way to have everything with you when you move from one machine to the next. This is reasonable and, if always done right, what you're doing should work. As for .Mac, it uses IMAP and should also work well. Which kind of problems were you experiencing?

rsync version 2.6.6 protocol version 29

I meant whether this is the version of rsync that comes with Tiger. I guess the answer is yes. You shouldn't use it, its broken -- Google for Tiger rsync or just search for rsync in either the Mac OS X Tiger or the Unix forums, e.g.:

http://discussions.apple.com/thread.jspa?threadID=404954

rsync -av

This is wrong.

For one thing, you're not telling rsync to take metadata into consideration. The Apple's version that comes with Tiger has an -E or --extended-attributes option for this, as do other alternative (better) implementations that try to fix it.

Second, you forgot to specify the --delete flag. This is necessary to (quoting rsync's man page) "delete any files on the receiving side that aren't on the sending side". In Tiger Mail, each message is stored as a separate file in the filesystem. If you tell Mail to delete a message, the corresponding file in the filesystem will be deleted as well. If this message was already present the last time you used Mail on another machine, it'll be there again after synching with that machine (because you're telling rsync to not propagate file deletions). Worse yet, the message will be there but any references to it in the HOME/Library/Mail/Envelope Index master index file and elsewhere would have been removed, and this is bad.

Another consequence of not using the --delete flag is that you're likely to have "orphaned" account folders in HOME/Library/Mail/ for any accounts you've deleted in Mail since you started to use rsync. For each mail account, Mail keeps a folder in HOME/Library/Mail/. The name of that folder begins with the account type, continues with the account user name, and ends with the incoming mail server, e.g. HOME/Library/Mail/POP-username@mail.server.com. When an account is removed in Mail > Preferences > Accounts, Mail removes the account from the preferences file HOME/Library/Preferences/com.apple.mail.plist and deletes the associated HOME/Library/Mail/POP-username@mail.server.com folder (and any mail within it). But if you later rsync HOME/Library/Mail/, without the --delete flag, against a backup that obviously still has the HOME/Library/Mail/POP-username@mail.server.com account folder, that folder won't be deleted in the backup, thus effectively leaving it there forever, just as if it had never been deleted, despite the account itself has indeed been removed from the preferences file.

So, to summarize, here's the least you should do to fix your problem:

1. Make a backup of your current Mail data to a safe place in case any of the following steps goes wrong.

2. Use a better implementation of rsync than the one provided by Apple. For your purposes, I recommend the one provided by Quinton Dolan. If you put it in /usr/bin, beware that it may be replaced by any future Apple Update; this may or may not be what you want. Alternatively, you may put it in /usr/local/bin, in which case you must ensure that the right version of rsync is called by your scripts.

3. Replace rsync -av with:

rsync -av --extended-attributes --delete

or just

rsync -avE --delete

in all your scripts.

Additionally, you may also want to clean up your existing Mail data:

1. Quit Mail.

2. Look for any "orphaned" account folders in HOME/Library/Mail/ and delete them. Do not delete anything you're not sure belongs to an already deleted mail account.

3. Read the article Some mailbox files used by Mac OS X 10.3 are not deleted after importing. This article applies not only to mailbox (.mbox) folders within HOME/Library/Mail/Mailboxes/, but also to any mailbox folders within each of the account specific folders ( HOME/Library/Mail/POP-username@mail.server.com). Basically, you can remove any files (not folders) found within any .mbox folder except Messages and Info.plist (the article says you can also remove Info.plist files within .mbox folders, and in fact you can, but Tiger Mail does use them and will recreate them if deleted, so there is no point in doing so).

4. Move the HOME/Library/Mail/Envelope Index file to the Trash. This will cause Mail to recreate that file and reindex everything again the next time it's launched. I don't recommend doing this without having done step 3 first.

Note: As you can see, your problem is no walk in the park and I might very well have made a mistake in what I've written. Don't hesitate to ask if there is anything you don't understand or feel comfortable with.

ps: what ya doin' in barcelona?

Well, that's where I live (in Martorell, actually, a town 27 Km away).

ii) the "-E" resource forks bit: no matter which
version of rsync I have been using, the "-E" version
always caused trouble.

The -E option is not just for preserving resource forks. It's also for preserving other "extended attributes" (i.e. metadata). I've seen people experiencing problems with the -E option reporting that those problems disappeared if Spotlight was disabled. Maybe the reason -E caused troubles to you is the same reason you're experiencing problems after running rsync without it, i.e. Spotlight is kicking one way or another.

I haven't experienced any problems with the version I recommended in my previous reply, except I cannot use if for incremental backups because it keeps unnecessarily copying the metadata part of many unmodified files (most of which don't have resource forks, BTW). Hmmm, now that I think of it, I have Spotlight disabled in my backup volume, I wonder if that could be the reason it works without a hitch for me...

Anyway, whatever version of rsync you use, be sure to not use the one currently provided by Apple, unless you don't mind losing the timestamp information of your files.

will try it out!

Be sure to also take a look at cleaning up your Mail database as I described in my previous reply, even if part of it is not applicable to your case.

doing what?

Aside from being in these forums? Not much, really... 🙂