Server 5.4 SpamAssassin Bayesian files growing at enormous rate

I confirm other reports of this problem that amavisd and spamassassin are incompatible with APFS volumes.

Moving the .spamassassin file to an HDFS+ volume fixes the issue:

sudo rm -fr /Library/Server/Mail/Data/scanner/amavis/.spamassassin

sudo mkdir -p /Volumes/Macintosh\ HD/Library/Server/Mail/Data/scanner/amavis/.spamassassin

sudo ln -s /Volumes/Macintosh\ HD/Library/Server/Mail/Data/scanner/amavis/.spamassassin /Library/Server/Mail/Data/scanner/amavis/.spamassassin

sudo chown -R _amavisd:_amavisd /Volumes/Macintosh\ HD/Library/Server/Mail/Data/scanner

sudo chmod 770 /Volumes/Macintosh\ HD/Library/Server/Mail/Data/scanner

sudo chown -h _amavisd:_amavisd /Library/Server/Mail/Data/scanner/amavis/.spamassassin

sudo chmod -h 770 /Library/Server/Mail/Data/scanner/amavis/.spamassassin

sudo /Applications/Server.app/Contents/ServerRoot/usr/libexec/spamassassin/sa_learn. pl

sudo ls -lH /Library/Server/Mail/Data/scanner/amavis/.spamassassin

total 2512

lrwxr-xr-x 1 _amavisd _amavisd 54 Feb 1 21:03 .spamassassin -> /Library/Server/Mail/Data/scanner/amavis/.spamassassin

-rw-------+ 1 root _amavisd 32768 Feb 1 21:26 bayes_seen

-rw-------+ 1 root _amavisd 1249280 Feb 1 21:26 bayes_toks

This also works with a spam/ham in the junkmail/notjunkmail folders.

Other mentions of this issue:

• Spamassassin & Amavisd borked today
• Server 5.4 SpamAssassin Bayesian files growing at enormous rate
• _amavisd hanging Server.app 5.4
• Server.app v5.5 amavisd and spamassassin filling hard drive, crashing server every night

Please file bug reports.

APFS fails with open source MTA tools, whether they’re part of the existing Server.app 5.6 or compiled from Macports.

The commands

/Applications/Server.app/Contents/ServerRoot/usr/sbin/postalias /tmp/aliases

sudo port install postfix

/opt/local/bin/newaliases -oA/tmp/aliases

all create an enormous aliases.db file.

It appears for now that macOS with APFS cannot be used as a mail server.

I’m going to look into a low power, OpenBSD-based server options, like one of these this fanless atom system:

• SUPERMICRO MBD-A1SRi-2758F-O Mini ITX Server Motherboard DDR3 1600/1333 - Newegg.com
• https://www.amazon.com/gp/product/B00HS4NLHA/

This solution could handle the PF firewall, DHCP, DNS, Apache, OpenVPN, all at multi Gb/s on the LAN, drawing maybe 10 Watts, and at a tiny fraction (perhaps a fifth) of the expected cost of the new Mac Pro’s promised this year.

I was planning on a new MP before the demise of Server.app, and I’d still prefer an all-Apple vertically integrated solution to mitigate the multiplatform maintenance, but this appears to be impossible on macOS.

I have this problem as well. Based on advice found elsewhere, I was simply deleting the .spamassasin directory, which worked for a while. My plan was to keep doing this until I had time to find a proper solution, but the machine eventually wouldn’t let me log in. Tried rebooting, but now even that fails. Tried booting into target disk mode and restore mode, but those don’t work either. Running OS X 11 if memory serves.

As I said, that’s what I was doing, but it’s only a temporary fix. The files would eventually reappear. And eventually, not only could I not ssh in, I couldn’t log in at the machine itself, under any account. And in my case the files took over 300G of space in a frightfully small amount of time. As the original poster said, seems like a pretty major bug.

FWIW, I wound up wiping the drive, and restoring from a time machine backup (which was tricky because I had to find a backup that wasn’t of the full disk), and then turning off everything but blacklist filtering.

The only complication was that somehow on one reboot (I’m guessing) Server started before the disk containing all the ‘network’ user accounts was mounted. It mounted as ‘Users 1’ which of course messed up everything. Deleted the Users directory (I assume) Server created, remounted the drive, and everything seems to be working smoothly since.

Worried the same thing will happen again next time I reboot.

The fix for me was:

1) make sure junkmail and nonjunkmail user accounts exist (I have them as local users)

2) allow sa_learn.pl and to run automatically (this should be 02:15 every day) and the output should be

sa_learn.pl: -- database synchronization complete

in in the file /Library/Logs/Mail/junkmail.log

3) check the following are in /Library/Server/Mail/Config/spamassassin/local.cf

bayes_auto_learn 1

use_bayes 1

time_limit 200

bayes_auto_expire 0

Stop and restart mail.

you may need all of these or just one.... my server started live as 10.3 so some things have moved on.

I no longer have this problem and i no longer have TIMEOUT of spamassassin.

I can confirm the problem still exists in High Sierra 10.13.3 using SpamAssassin 3.4.1 on Perl 5.24 installed via MacPorts.

Trying to analyze the problem I noticed that the same situation occurs when running 'make test' for Mail::SpamAssassin the bayesdbm.t. My guess is that either SpamAssassin's Bayesdbm.pm module or one of it's dependencies is the culprit.

I've had the issue since upgrading to 10.13. I just disabled filtering until now.

After trying everything in this thread (which didn't help) this fixed the issue for me!

https://www.pingle.org/2011/02/10/rapidly-growing-bayes_toks

sa-learn --backup > bayes_backup.txt

sa-learn --restore bayes_backup.txt

Mac OS 10.13.3

Server 5.5

APFS

Delete your .spamassasin folder or files to reclaim your disk space.

/Library/Server/Mail/Data/scanner/amavis/.spamassassin

Enjoy!

I'm afraid it didn't work for me.

Backup phase is successful but the restoration phase starts accumulating a bayes_toks.convertXXXXX file. I lost fate and killed the restore at 32GB (my bayes_toks db is 5MB and bayes_seen 10MB). 😟

Tried both with and without deleting the bayes_toks and bayes_seen db's in between backup and restore.

I have reported the issue to SA Bugzilla but I'm not holding my breath. Please do contribute to the ticket, though, if you feel you have additional information.

7538 – Bayes_toks grow indefinitely on macOS High Sierra when using APFS

Hi, Y'all,

I had two partitions on the same disk, one APFS and the other HFS. I typically do an update to macOS and macOS Server by shutting down all services and then using Carbon Copy Cloner to make a full backup of the drive to the other drive. I had a failed/stalled update (still don't know why) and ended up rebooting off the clone on the APFS partition. I made a successful update and all was well. Until it wasn't.

Shortly after that, my startup disk complained that it was getting full (like, a few megabytes free), the machine was running slowly, and I had no idea what was taking up all that space. Folder sizes were ordinary, but I was able to download Omni DiskSweeper et voilà! I saw that I had numerous bayes_toks.expire files of enormous (tens of GBs) size each in /var/root/.spamassassin. I deleted them, deleted the bayes_toks files in /Library/Server... everything. Did the sa_learn step... and they would come back less than 24 hours later.

After finding this thread, I investigated and discovered I'd rebooted to the APFS volume and that might be the source of the problem. I shut down services, made a CCC clone to the HFS partition, rebooted, and since then (two months ago), I have not seen my bayes* files grow in size to any abnormal size. The files in /var/root/.spamassassin haven't been touched since that day or so, and the files in /Library/Server... are of a reasonable size, ≈1MB for bayes_toks and 170K for bayes_seen.

Having changed nothing else that I am aware of, I must conclude that APFS is causing the problem.

If I were brave, I'd CCC to the APFS volume and observe, but there's a limit to my patience.

/Bill

This is a known issue with APFS, and there are several duplicate bug reports.

This bugs impact isn’t limited to spamassassin. You can also tickle it with the newaliases command on an APFS volume:

sudo newaliases -oA/Library/Server/Mail/Config/postfix/aliases

creates an enormous aliases.db file.

The only workaround now is to host these files on an HFS volume.

Symlinks to an HFS volume are seemingly a good fix. I tried to symlink the entire /Library/Server/Mail/Data/scanner folder, but clamav refused to work over a symlink.

The postfix databases also had to be shifted. Here are the symlinks I created pointing to my HFS RAID from the APFS boot volume:

office:Mail root# ls -al /Library/Server/Mail/Data/scanner/amavis/ total 64 drwxr-x--- 16 _amavisd _amavisd 512 26 Mar 21:39 . drwxr-xr-x 6 root wheel 192 24 Mar 14:52 .. lrwxr-xr-x 1 _amavisd _amavisd 70 26 Mar 08:27 .spamassassin -> /Volumes/Promise/Library/Server/Mail/Data/scanner/amavis/.spamassassin -rw-r----- 1 _amavisd _amavisd 0 27 Mar 09:29 amavisd.lock -rw-r----- 1 _amavisd _amavisd 6 26 Mar 14:30 amavisd.pid srwxr-x--- 1 _amavisd _amavisd 0 26 Mar 08:36 amavisd.sock srw-rw---- 1 _amavisd _amavisd 0 26 Mar 08:36 clamd -rw-rw-r-- 1 _amavisd _amavisd 6 26 Mar 08:36 clamd.pid drwxr-x--- 8 _amavisd _amavisd 256 26 Mar 14:30 db -rw-r----- 1 _amavisd _amavisd 160 2 May 2015 local_domains -rw-r----- 1 _amavisd _amavisd 16384 2 May 2015 local_domains.db drwxr-x--- 131 _amavisd _amavisd 4192 27 Mar 09:30 tmp drwxr-x--- 2 _amavisd _amavisd 64 30 Apr 2015 user_prefs drwxr-xr-x 3 _amavisd _amavisd 96 24 Mar 11:17 var drwxr-x--- 2 _amavisd _amavisd 64 28 Apr 2015 virusmails -rw-r--r-- 1 _amavisd _amavisd 1 28 Apr 2015 whitelist_sender office:Mail root# ls -al /Library/Server/Mail/Data/ total 0 drwxr-xr-x 16 root wheel 512 26 Mar 21:40 . drwxr-xr-x 4 root wheel 128 26 Mar 08:23 .. drwxr-xr-x 3 root wheel 96 28 Apr 2015 amavisd drwxrwxr-x 3 _dovecot mail 96 25 Mar 11:31 attributes drwxrwx--- 4 _postfix mail 128 26 Mar 12:27 db drwxr-xr-x 3 root wheel 96 20 Sep 2015 dovecot drwxr-xr-x 6 nobody mail 192 24 Mar 11:14 gldb drwxrwx--- 2 _dovecot mail 64 20 Sep 2015 keys lrwxr-xr-x 1 root wheel 52 26 Mar 08:12 listserver -> /Volumes/Promise/Library/Server/Mail/Data/listserver drwxrwxr-x 72 _dovecot mail 2304 27 Mar 02:02 mail drwxr-x--- 6 _postfix mail 192 25 Mar 09:34 mta drwxrwxr-x 40 _dovecot mail 1280 22 Feb 14:18 rules drwxr-xr-x 6 root wheel 192 24 Mar 14:52 scanner drwxrwx--- 3 _dovecot mail 96 12 Mar 16:48 shared drwxrwx--- 5 _dovecot mail 160 20 Sep 2015 sieve drwxr-xr-x 16 root wheel 512 28 Apr 2015 spool office:Mail root# ls -al /Library/Server/Mail/Config/ total 16 drwxr-xr-x 11 root wheel 352 27 Mar 03:42 . drwxr-xr-x 4 root wheel 128 26 Mar 08:23 .. -rw-r--r-- 1 root wheel 5037 27 Mar 03:42 MailServicesOther.plist drwxr-xr-x 3 root wheel 96 26 Mar 08:36 amavisd drwxr-xr-x 4 root wheel 128 8 Mar 2016 clamav drwxr-xr-x 14 root wheel 448 26 Mar 08:36 dovecot drwxr-xr-x 3 root wheel 96 28 Apr 2015 listserver lrwxr-xr-x 1 root wheel 51 26 Mar 08:23 postfix -> /Volumes/Promise/Library/Server/Mail/Config/postfix drwxr-xr-x 50 root wheel 1600 26 Mar 08:04 postfix.actual drwxr-xr-x 5 root wheel 160 28 Apr 2015 resources drwxr-xr-x 13 root wheel 416 24 Mar 11:01 spamassassin

Unfortunately, /Library/Server/Mail/Data/scanner/amavis/localdomains.db is still a problem, but I can't symlink its parent. Might just have to keep my eye on that one. If anyone knows how to overcome clamav working over a symlink, I can symlink the entire scanner directory to fix that.