_amavisd hanging Server.app 5.4

Question

Level 1

45 points

_amavisd hanging Server.app 5.4

Hey All,

I had the spamassassin eating space problem with the added mystery of the GUI not displaying accurate usage.

The GUI completely ignored the the /Library/Server/Mail/Data/scanner/amavis/.spamassassin file size in tally up the disk usage.

Once I cleared that upped I still have problems with amavisd becoming unresponsive?.

Postfix shows connections are being refused. The Server.app GUI shutting down and starting up amavisd does not kill the perl5.18 processes owned by _amavisd using 98.8 percent of the CPU and the oldest in the activity monitor.

If I kill the process using activity monitor everything returns to normal until the next hang.

I have found the /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.a pple.amavis.amavisd.plist and I'm considering try to use that to unload when amavisd becomes stuck.

Any troubleshooting tips appreciated.

Ben

Posted on Nov 5, 2017 6:14 AM

Reply

Answer 1

detourdog Author

Level 1

45 points

Nov 5, 2017 7:24 AM in response to detourdog

I finally got around to editing my

/Library/Server/Mail/Config/amavisd/amavisd/conf

I changed my

$max servers =2;

to

$max servers=18;

This controls the number of pre-forked children which looks to be exact the constraint I'm running against.

I'm pretty optimistic about this fix at least providing larger amounts of uptime if not resolving the issue.

Thanks,

Ben

Reply

Answer 2

detourdog Author

Level 1

45 points

Nov 6, 2017 4:28 AM in response to detourdog

Addding more servers only increased the amount of time before I need to kill old amavisd-new processes.

I'm still digging into the issue and think it must have to do with incoming email tieing up the amavisd-new processes.

Any comments considered.

Ben

Reply

Answer 3

detourdog Author

Level 1

45 points

Nov 6, 2017 5:45 AM in response to detourdog

Ok, I think I have a good message from the log file.

perl5.18 reprots in the system log that too many groups requested (2147483647). Can cause performance issues when network directories are involved.

I was able to trigger this message by turning off port 25 access on my outside firewall. I was getting some mail but not enough for me to be confident that I was getting all the mail traffic I needed.

As soon as I turned out port 25 traffic the system log exploded with the above message.

Repeated once for each server instance set in $max servers.

I'm going to continue toying until I think can put a stop to this.

Thanks,

Ben

Reply

Answer 4

detourdog Author

Level 1

45 points

Nov 6, 2017 6:49 AM in response to detourdog

One theory dies and a new one is born.

I was trying to figure out how to limit the number of protocols I would accept on port 25. It seems that TLS connections begin on 25 so it is better on then off.

In Server.app 5.4 my authentication method pull down in Mail was set to automatic.

I think this setting allows anyone on the internet to try anything they want on port 25.

I opted for the Open Directory when limits some of the traffic options on port 25.

This seems to have a quicker dropping of random connections trying to authenticate on port 25.

I'm seeing a much better response time and very few _amavisd owned perl5.18 processes being utilized.

If this appears to clear things up I will reduce my $max processes to 18.

Thanks,

Ben

Reply

Answer 5

i-Dono

Level 1

4 points

Nov 10, 2017 12:42 AM in response to detourdog

Hi detourdog!

I have the same problem. I did all the things you mentioned. But nothing help finally.

But I believe the root cause is the bayes_toks file (/Library/Server/Mail/Data/scanner/amavis/.spamassassian/bayes_toks). It growth up to 30GB in my case and I believe that made the amavisd task running all the time.

May be there is a misconfiguration in our case (see https://forums.cpanel.net/threads/bayes_seen-and-bayes_toks-eating-up-a-lot-of-d isk-space.79509/ and https://www.pingle.org/2011/02/10/rapidly-growing-bayes_toks). In the second link somebody wrote something about "auto learn" and "auto expiration" (see below quote) properties which may not be okay in our case, but I could not find them for changing.

Quote:

I checked all the usual things: auto learning was off, auto expiration was off, the permissions and user were set correctly, and so on. And yet it grew, constantly and swiftly.

Reply

Answer 6

detourdog Author

Level 1

45 points

Nov 10, 2017 6:24 AM in response to i-Dono

I was holding back one last tweak that may have fixed it. I will let you know what it is but you have to report back if it makes a difference either way.

around line number 196 in my /Library/Server/Mail/Config/amavisd/amavisd.conf

$bad_header_quarantine_method = D_DISCARD;

to

$bad_header_quarantine_method = undef

I wanted to make that change intutiatively for a day or 2 and the problem stopped shortly after I did make that change.

The comcast outage also ended about the time things started working so I thought it may have been related to that.

Please let me know if if makes a difference.

Reply

Answer 7

Nov 19, 2017 10:21 AM in response to detourdog

Hi, I had this problem as well. I think that APFS confuses spamassassin. What I did was to create a .spamassassin directory on a HFS+ drive, then symlink /Library/Server/Mail/Data/scanner/amavis/.spamassassin/ to that directory. Of course I did that with mail server turned off. I pre-populated the symlinked directory with files I restored from a pre-High Sierra spamassassin.

I suspect the mismatch in size has to do with APFS's support for sparse files. For example, if I did an "sa-learn --backup" from my Sierra spamassassin data, then "sa-learn --restore" to the HFS+ drive, it completed in a few seconds. If I did the same thing with output to APFS, after ~30 minutes it had created a 30 TB file on a ~200 GB filesystem, which was mostly empty before and after the file was created.

Reply

Answer 8

detourdog Author

Level 1

45 points

Nov 19, 2017 10:25 AM in response to Brian Landy

I will definitely give it a go.

This is a very plausible scenario.

The crazy thing is how intermittent it is for me. I can go days without a problem...

I will make this change and see if the problem is gone for an entire 14 days I would call it solved.

Thanks,

Ben

PS. do you know anything about $penpals

Reply

Answer 9

i-Dono

Level 1

4 points

Dec 11, 2017 12:26 AM in response to Brian Landy

I also did a symlink of /Library/Server/Mail/Data/scanner/amavis/.spamassassin/ to an HFS+ Volume because my system drive is APFS formatted. Since 20th November the bayes_toks stayed at 16kB size. Now extrem growing anymore and the whole system is fast and stable!

I believe the APFS makes the trouble....

Reply

Answer 10

detourdog Author

Level 1

45 points

Dec 13, 2017 2:58 AM in response to i-Dono

I agree that the symlink makes a big difference and I could see immediately in Activity Monitor that the processes owned by _amavid %CPU dropped from spikes of 3-6% to almost 0 always.

I also found that the file :

/Library/Server/Mail/Data/scanner/amavis/.spamassasin

Could not reliable report it's size to the OS while on an APFS formatted SSD. This caused all sorts of normal diagnosis difficult using unit tools and conflicting GUI size reports. I formatted a 32 GB USB HFS+ formatted thumb drive move the .spamassassin file there with a symlink to /Library/Server/Mail/Data/scanner/amavis/.spamassassin. I saw an immediate drop in CPU demand and have had solid uptime for amavisd.

I have had virtual memory eat-up 32 GB of drive space twice and in the past 24 hours I have seen my original amavisd symptoms come back.

I would like to close the issue but I will call it a very good idea and maybe the cure. I want to run through my config again.

I'm really glad Brian suggested this but I'm not sure it has fixed my issue. Has anyone else run out of VM or seen a re-occurence?

Thanks,

Ben

Reply

Answer 11

detourdog Author

Level 1

45 points

Dec 18, 2017 5:48 AM in response to detourdog

I have my amavisd slowing to halt problem and putting a symlink to .spamassassin on an HFS+ drive has definitely seems to have helped, but...

It could be that there is another thing going on. My problem returned after almost 2 week of not occurring.

I seem to have cleared it up this time by clearing the contents of /Library/Server/Mail/scanner/amavis/.spamassassin.

Which fundamentally is why I did when I did the symlink between /Library/Server/Mail/scanner/amavis/.spamassassin and my HFS+ .spamassassin/ folder.

To summarize I seem to be having a problem with a corrupted file in /Library/Server/Mail/scanner/amavis/.spamassassin/ which only develops overtime and clears up if a clear out that folder.

I will keep this going.

Reply