Spamassassin Junkmail and Notjunkmail Learning in Mountain Lion

Question

Level 1

41 points

Spamassassin Junkmail and Notjunkmail Learning in Mountain Lion

Like many others <https://discussions.apple.com/thread/4181320?start=0&tstart=0>, I wondered what happened to spamassassin and its Bayesian learning mailboxes "junkmail" and "notjunkmail" in OS X Server. Perhaps Apple had automated this process? There certainly isn't any documentation. I didn't think much of it until spam began falling into my server's INBOX, whereupon looking for a fix I discovered that spamassassin learning capabilities still work in Mountain Lion OS X Server, but they aren't automated in any way.

Here's how to train your server to detect the junk mail you receive, beyond the appropriate spam filter and greylist settings Server.app>Mail.

First, confirm that your serveradmin settings know about the spamassassin Bayesian learning mailboxes "junkmail" and "notjunkmail":

sudo serveradmin settings mail:imap | grep junk

mail:imap:junk_mail_userid = "junkmail"

mail:imap:not_junk_mail_userid = "notjunkmail"

Then use Server.app to add the network accounts "junkmail" and "notjunkmail" and edit access so that these accounts only have access to the Mail service. Add these accounts to Mail so that you (and other trusted users) have access to them.

Then dump all your junkmail from your server accounts and other cloud email accounts into the "junkmail" mailbox. Move any messages that aren't junk but marked as such into the mailbox "notjunkmail".

Finally,

sudo /Applications/Server.app/Contents/ServerRoot/usr/libexec/spamassassin/learn_jun k_mail.sh

This will train spamassassin based on the content of these mailboxes, and hopefully you will begin to see amavisd properly mark junkmail as "** JUNK MAIL **" in the subject line.

Mac mini Server (Mid 2010), OS X Mountain Lion (10.8.2), Lion Server, EyeTV HD, Turbo.264 HD

Posted on Nov 4, 2012 2:40 AM

Reply

Answer 1

Kostas B

Level 1

94 points

Nov 4, 2012 6:03 AM in response to essandess

This is great info, thanks for sharing.

2 questions:

1. How do you set this accounts as "public folders"? Or you just instruct users to add these accounts in their Mail client?

2. Does "learning" and removing false spam marking really works?

Best regards

Kostas

Reply

Answer 2

essandess Author

Level 1

41 points

Nov 4, 2012 10:40 AM in response to Kostas B

These are IMAP account names, not public folders (even though they do exist as folders within the guts of /Library/Server/Mail/Data, you have to let IMAP worry about that stuff). You can either:

1. Instruct users to Mail>Message>Send Again [Cmmand-Shift-D], then send the junk/notjunk to these respective accounts.

2. Distribute the password(s) for the junkmail and notjunkmail accounts, allowing users to add them and simply drag and drop directly, or add a rule do redirect junk.

Yes, Bayes Rule really works. I see "*** JUNK MAIL ***" marked email notifications as I type this from the same spammer that was making it through unmarked before (a pretty good feat these days!). This junk is automatically directed into my personal Junk Mail folder. I'll added them to junkmail later, or write a rule to do this automatically.

Spamassassin has script to unlearn mistakes, though I've never used it. Also, I believe that Snow Leopard Server automatically scraped the contents of the junkmail and notjunkmail accounts every week, making the last script above unnecessary—we have not been told whether this feature exists in Mountain Lion, thou it wouldn't be hard ton search for any daemon that called the script learn_junk_mail.sh .

Reply

Answer 3

Nov 9, 2012 4:36 PM in response to essandess

Hi,

I discovered this post and tried the command you suggested for training junkmail, however I got an error message instead:

server:~ admin$ sudo /Applications/Server.app/Contents/ServerRoot/usr/libexec/spamassassin/learn_jun k_mail.sh

Password:

grep: /Applications/Server.app/Contents/ServerRoot/etc/dovecot/partition_map.conf: No such file or directory

Can't determine mail store path from /Applications/Server.app/Contents/ServerRoot/etc/dovecot/partition_map.conf

How can I fix this?

Thanks

Jefff

Reply

Answer 4

essandess Author

Level 1

41 points

Nov 9, 2012 5:10 PM in response to Jeff Hargrove

Curious. I can just show you my server's setup, and perhaps if you replicate the problem will go away.

ls -l /Applications/Server.app/Contents/ServerRoot/etc/dovecot/partition_map.conf

lrwxr-xr-x 1 root wheel 54 Oct 28 22:41 /Applications/Server.app/Contents/ServerRoot/etc/dovecot/partition_map.conf -> /Library/Server/Mail/Config/dovecot/partition_map.conf

ls -l /Library/Server/Mail/Config/dovecot/partition_map.conf

-rw-r--r-- 1 root wheel 39 Sep 22 01:54 /Library/Server/Mail/Config/dovecot/partition_map.conf

cat /Library/Server/Mail/Config/dovecot/partition_map.conf

default:/Library/Server/Mail/Data/mail

Reply

Answer 5

essandess Author

Level 1

41 points

Nov 9, 2012 5:13 PM in response to Jeff Hargrove

P.S. You fixed my cut-and-paste typo, right? The correct command is:

sudo /Applications/Server.app/Contents/ServerRoot/usr/libexec/spamassassin/learn_jun k_mail.sh

Reply

Answer 6

essandess Author

Level 1

41 points

Nov 9, 2012 5:15 PM in response to Jeff Hargrove

Wow. This editor keeps adding that extra space in the word learn_junk_mail.sh. Never mind.

Reply

Answer 7

Nov 9, 2012 5:29 PM in response to essandess

Yes, I did fix the typo.

However following your commands, I still get an absent partition_map.conf file:

server:~ admin$ ls -l /Applications/Server.app/Contents/ServerRoot/etc/dovecot/partition_map.conf

ls: /Applications/Server.app/Contents/ServerRoot/etc/dovecot/partition_map.conf: No such file or directory

server:~ admin$ ls -l /Library/Server/Mail/Config/dovecot/partition_map.conf

-rw-r--r-- 1 root admin 39 7 nov 08:22 /Library/Server/Mail/Config/dovecot/partition_map.conf

server:~ admin$ cat /Library/Server/Mail/Config/dovecot/partition_map.conf

default:/Library/Server/Mail/Data/mail

server:~ admin$

Is it the same file in both locations? Can I copy it into /Applications/Server.app/Contents/ServerRoot/etc/dovecot ?

Reply

Answer 8

essandess Author

Level 1

41 points

Nov 9, 2012 5:36 PM in response to Jeff Hargrove

You have to issue these commands to replicate what I see on my server:

sudo ln -s /Library/Server/Mail/Config/dovecot/partition_map.conf /Applications/Server.app/Contents/ServerRoot/etc/dovecot/partition_map.conf

sudo chown root:wheel /Applications/Server.app/Contents/ServerRoot/etc/dovecot/partition_map.conf

sudo chmod 755 /Applications/Server.app/Contents/ServerRoot/etc/dovecot/partition_map.conf

Jobs knows why that symbolic link didn't come with your Server.app.

Reply

Answer 9

Nov 9, 2012 5:51 PM in response to essandess

haha, Yes I agree. With each clean installation, nothing is the same!

Well it seems to have worked! Does this look right to you?

server:~ admin$ sudo /Applications/Server.app/Contents/ServerRoot/usr/libexec/spamassassin/learn_jun k_mail.sh

Learning what is junk mail

--- sudo -u _amavisd /Applications/Server.app/Contents/ServerRoot/usr/bin/sa-learn --dbpath /Library/Server/Mail/Data/scanner/amavis/.spamassassin --spam --no-sync < /Library/Server/Mail/Data/mail/45D0C5F6-DBC3-48A0-AAF9-7D69F79E0FEC/cur/1352283 956.M74829P7112.server.jeffhargrove.com,S=50277,W=50949:2,Sa

--- sudo -u _amavisd /Applications/Server.app/Contents/ServerRoot/usr/bin/sa-learn --dbpath /Library/Server/Mail/Data/scanner/amavis/.spamassassin --spam --no-sync < /Library/Server/Mail/Data/mail/45D0C5F6-DBC3-48A0-AAF9-7D69F79E0FEC/new/1352293 084.M102587P973.server.jeffhargrove.com,S=49035,W=50239

--- sudo -u _amavisd /Applications/Server.app/Contents/ServerRoot/usr/bin/sa-learn --dbpath /Library/Server/Mail/Data/scanner/amavis/.spamassassin --spam --no-sync < /Library/Server/Mail/Data/mail/45D0C5F6-DBC3-48A0-AAF9-7D69F79E0FEC/new/1352308 002.M789666P4300.server.jeffhargrove.com,S=6250,W=6422

--- sudo -u _amavisd /Applications/Server.app/Contents/ServerRoot/usr/bin/sa-learn --dbpath /Library/Server/Mail/Data/scanner/amavis/.spamassassin --spam --no-sync < /Library/Server/Mail/Data/mail/45D0C5F6-DBC3-48A0-AAF9-7D69F79E0FEC/new/1352371 908.M238607P32768.server.jeffhargrove.com,S=1362,W=1391

--- sudo -u _amavisd /Applications/Server.app/Contents/ServerRoot/usr/bin/sa-learn --dbpath /Library/Server/Mail/Data/scanner/amavis/.spamassassin --spam --no-sync < /Library/Server/Mail/Data/mail/45D0C5F6-DBC3-48A0-AAF9-7D69F79E0FEC/new/1352495 615.M32695P46666.server.jeffhargrove.com,S=16231,W=16452

--- sudo -u _amavisd /Applications/Server.app/Contents/ServerRoot/usr/bin/sa-learn --dbpath /Library/Server/Mail/Data/scanner/amavis/.spamassassin --spam --no-sync < /Library/Server/Mail/Data/mail/45D0C5F6-DBC3-48A0-AAF9-7D69F79E0FEC/new/1352495 629.M13461P46677.server.jeffhargrove.com,S=19331,W=19592

--- sudo -u _amavisd /Applications/Server.app/Contents/ServerRoot/usr/bin/sa-learn --dbpath /Library/Server/Mail/Data/scanner/amavis/.spamassassin --spam --no-sync < /Library/Server/Mail/Data/mail/45D0C5F6-DBC3-48A0-AAF9-7D69F79E0FEC/new/1352495 648.M683301P46689.server.jeffhargrove.com,S=13378,W=13563

--- sudo -u _amavisd /Applications/Server.app/Contents/ServerRoot/usr/bin/sa-learn --dbpath /Library/Server/Mail/Data/scanner/amavis/.spamassassin --spam --no-sync < /Library/Server/Mail/Data/mail/45D0C5F6-DBC3-48A0-AAF9-7D69F79E0FEC/new/1352495 675.M29820P46709.server.jeffhargrove.com,S=13001,W=13180

Learning what is not junk mail

find: /Library/Server/Mail/Data/mail/5A73D415-CF7B-4C2F-AAEF-FF2F8536493E/cur: No such file or directory

find: /Library/Server/Mail/Data/mail/5A73D415-CF7B-4C2F-AAEF-FF2F8536493E/new: No such file or directory

I don't like the last two lines, though!

Reply

Answer 10

essandess Author

Level 1

41 points

Nov 9, 2012 6:07 PM in response to Jeff Hargrove

That's it! All those crazy file names are imap email files in your junkmail account. The last two lines are just because you've never stuck anything into the notjunkmail account, and imap hasn't created these directories yet. That will go away if you ever use notjunkmail.

By the way, I've been getting (apparently) irrelevant errors about a missing config file in my ~/.spamassassin directory, errors that I don't see in your output. Would you please show us the result of this command on your server?

ls -ld ~/.spamassassin

username wheel 136 Apr 25 2011 /Users/username/.spamassassin

ls -lR ~/.spamassassin

total 2520

-rw-rw---- 1 username wheel 49152 Apr 25 2011 bayes_seen

-rw-rw---- 1 username wheel 1241088 Apr 25 2011 bayes_toks

By the way, it was pretty funny watching the Republicans turn themselves into Bayes' Theorem denialists this past month.

Reply

Answer 11

Nov 9, 2012 6:22 PM in response to essandess

Great! Thanks for your help!

Ok, this is weird ... I tried those commands and I'm getting no such file or directory response! Should I be worried?

Reply

Answer 12

essandess Author

Level 1

41 points

Nov 9, 2012 6:27 PM in response to Jeff Hargrove

Ahhh. That's it --- "rm -fr ~/.spamassassin". Thanks -- that makes my errors go away. No, you shouldn't worry -- I had a superfluous personal ~/.spamassassin directory from a prior install or just messing around. Deleting it is the right thing because as you can see above, OS X server has its own special place for the Bayes' database.

Reply

Answer 13

Nov 9, 2012 6:30 PM in response to essandess

I didn't do much, but was glad I could help a little after all the help you gave me!

By the way, could the absence of that symbolic link affect spam being quarantined to a mail box? I'm having an issue where spam is not quarantined to the address I gave it.

Reply

Answer 14

essandess Author

Level 1

41 points

Nov 9, 2012 7:13 PM in response to Jeff Hargrove

I ran the find command to see if Apple uses learn_junk_mail.sh automatically. It isn't defaulted, but they've included all the necessary hooks in a plist file:

find /Applications/Server.app -type f -exec fgrep -l learn_junk_mail.sh {} ';'

/Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.ap ple.salearn.plist

/Applications/Server.app/Contents/ServerRoot/usr/share/servermgrd/bundles/server mgr_mail.bundle/Contents/MacOS/servermgr_mail

These commands will set up your Server so that Spamassassin's Bayesian junk mail training occurs at 2:15 am every night:

sudo cp /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.a pple.salearn.plist /System/Library/LaunchDaemons

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.salearn.plist

sudo launchctl list | grep salearn

- 0 com.apple.salearn

About quarantining, this is independent of training, so the lack of that symbolic linlk shouldn't have an effect.

Also,

Mail>Preferences>Rules> Add Rule, Spamassassin, If <Message is Junk Mail>, then <Move Message> to mailbox <junkmail>.

Aside from manually hitting the thumbs down on an email, that should completely automate the junkmail workflow on OS X Server.

Reply

Answer 15

Nov 9, 2012 7:20 PM in response to essandess

This is really helpful!

I was reading in Apple's advanced server admin manual that it was a default that the server will train junk mail at 2:15. Another let down ...

I think you should be writing the unofficial manual for all of us novices. 😀

Any chance I could get your help about my quarantine problem? I think it is trying to deliver to the quarantine address, but fails, but don't know why.

Again thanks for your time is getting spam training set up.

Cheers

Jeff

Reply