Mail bounced - Command time limit exceeded

Question

Mail bounced - Command time limit exceeded

Recently I've been getting some mail bounced randomly from users on my server to other users on my server.
The bounce states:

Command time limit exceeded: "/usr/bin/cyrus/bin/deliver"
Action: failed
Status: 5.0.0
Diagnostic-Code: X-Postfix; Command time limit exceeded: "/usr/bin/cyrus/bin/deliver"

Mail Service running on XServe, Mail Store on XServe RAID connected directly to the XServe via the fiber channel.

Intel XServe, Mac OS X (10.4.11)

Posted on Oct 22, 2008 9:25 AM

Reply

Answer 1

pterobyte

Level 6

11,104 points

Oct 22, 2008 10:37 AM in response to tibor.moldovan

Check mailaccess.log and system.log for the timeframe.
See if Cyrus hangs or in case you use OD, OD was unresponsive.

Reply

Answer 2

Oct 22, 2008 10:49 AM in response to pterobyte

I do use OD, but I'm not seeing that it's being unresponsive.
The mail works fine for about 5 minutes after a reboot, and then it stops delivering.

The server has 4 GB of RAM, most of it seems unused.

Thank you for the quick reply!

Console.log:
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:29:59 collaborate1 master[2141]: process 3558 exited, signaled to death by 8
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:29:59 collaborate1 master[2141]: process 3559 exited, signaled to death by 8
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:29:59 collaborate1 master[2141]: process 3560 exited, signaled to death by 8
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:30:00 collaborate1 master[2141]: process 3561 exited, signaled to death by 8
Oct 22 12:30:00 collaborate1 imaps[3562]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: Cannot allocate memory
Oct 22 12:30:00 collaborate1 imaps[3562]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: cyrusdb error
Oct 22 12:30:00 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:30:00 collaborate1 master[2141]: process 3562 exited, signaled to death by 8
Oct 22 12:30:00 collaborate1 imaps[3563]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: Cannot allocate memory
Oct 22 12:30:00 collaborate1 imaps[3563]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: cyrusdb error
Oct 22 12:30:00 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:30:00 collaborate1 imaps[3564]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: Cannot allocate memory
Oct 22 12:30:00 collaborate1 imaps[3564]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: cyrusdb error

System.log:
Oct 22 12:29:59 collaborate1 imaps[3560]: starttls: TLSv1 with cipher AES128-SHA (128/128 bits new) no authentication
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:29:59 collaborate1 master[2141]: process 3560 exited, signaled to death by 8
Oct 22 12:29:59 collaborate1 imaps[3561]: TLS server engine: cannot load CA data
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:29:59 collaborate1 imaps[3561]: starttls: TLSv1 with cipher AES128-SHA (128/128 bits reused) no authentication
Oct 22 12:30:00 collaborate1 master[2141]: process 3561 exited, signaled to death by 8
Oct 22 12:30:00 collaborate1 imaps[3562]: DBERROR db4: Logging region out of memory; you may need to increase its size
Oct 22 12:30:00 collaborate1 imaps[3562]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: Cannot allocate memory
Oct 22 12:30:00 collaborate1 imaps[3562]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: cyrusdb error
Oct 22 12:30:00 collaborate1 imaps[3562]: TLS server engine: cannot load CA data
Oct 22 12:30:00 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:30:00 collaborate1 imaps[3562]: starttls: TLSv1 with cipher AES128-SHA (128/128 bits new) no authentication
Oct 22 12:30:00 collaborate1 master[2141]: process 3562 exited, signaled to death by 8
Oct 22 12:30:00 collaborate1 imaps[3563]: DBERROR db4: Logging region out of memory; you may need to increase its size
Oct 22 12:30:00 collaborate1 imaps[3563]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: Cannot allocate memory
Oct 22 12:30:00 collaborate1 imaps[3563]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: cyrusdb error
Oct 22 12:30:00 collaborate1 imaps[3563]: TLS server engine: cannot load CA data

Mailaccess.log:
Oct 22 12:29:59 collaborate1 master[2141]: process 3556 exited, signaled to death by 8
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:29:59 collaborate1 master[2141]: process 3557 exited, signaled to death by 8
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:29:59 collaborate1 master[2141]: process 3558 exited, signaled to death by 8
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:29:59 collaborate1 master[2141]: process 3559 exited, signaled to death by 8
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:29:59 collaborate1 master[2141]: process 3560 exited, signaled to death by 8
Oct 22 12:29:59 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:30:00 collaborate1 master[2141]: process 3561 exited, signaled to death by 8
Oct 22 12:30:00 collaborate1 imaps[3562]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: Cannot allocate memory
Oct 22 12:30:00 collaborate1 imaps[3562]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: cyrusdb error
Oct 22 12:30:00 collaborate1 master[2141]: IMAP connection limit reached. connection limit: 280 current connections: 283
Oct 22 12:30:00 collaborate1 master[2141]: process 3562 exited, signaled to death by 8
Oct 22 12:30:00 collaborate1 imaps[3563]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: Cannot allocate memory
Oct 22 12:30:00 collaborate1 imaps[3563]: DBERROR: opening /Volumes/MailRAID1/Mail/mail-database/tls_sessions.db: cyrusdb error

Message was edited by: tibor.moldovan

Reply

Answer 3

Oct 22, 2008 10:58 AM in response to pterobyte

OK, I've also noticed this in console log, shortly after reboot:

Console.log:
Oct 22 12:56:58 collaborate1 DirectoryService[88]: Potential VM growth in DirectoryService since client PID: 0, has 525 open references when the warning limit is 500.
Oct 22 12:57:04 collaborate1 DirectoryService[88]: Potential VM growth in DirectoryService since client PID: 0, has 550 open references when the warning limit is 500.
Oct 22 12:57:04 collaborate1 DirectoryService[88]: Potential VM growth in DirectoryService since client PID: 0, has 550 open references when the warning limit is 500.

Does this mean that my OD master is unresponsive?

Reply

Answer 4

pterobyte

Level 6

11,104 points

Oct 22, 2008 11:11 AM in response to tibor.moldovan

Looks like your IMAP database has issues. As a result, IMAP connections fail/are dropped. The mail clients try to establish a new connection each time this occurs, thus rapidly reaching the connection limits.

The memory allocation it refers to is not RAM, but the buffer sizes of BerkeleyDB (the underlying database).

Before any drastic measures, you should start to check if there is some broken mail client on your network. Don't know how many users you have, but hitting those limits within minutes of rebooting seems a but fast.

After that, you should start analysing your environment and fine tune it. Depending on number of users, connection, mail traffic, etc, you may need to adjust settings.

Reply

Answer 5

jaydisc

Level 4

1,464 points

Oct 22, 2008 7:04 PM in response to tibor.moldovan

Looks like you ran out of connections and RAM. I had a similar problem. One user had a dead imap connection. You can determine this by shutting down the mail server and running:

ps -axx | grep imapd

If you get any results, you have a dead imapd process and will need to restart (most likely forcibly) your server.

Reply

Answer 6

pterobyte

Level 6

11,104 points

Oct 23, 2008 3:17 AM in response to jaydisc

ps -axx | grep imapd will list all imapd processes. Even the ones that are running and sleeping.
The ones to watch out for are the ones with a U or a Z in front.

Reply

Answer 7

Oct 23, 2008 8:04 AM in response to jaydisc

OK, good to know. I'll take a look at those.
Does it matter that every user seems to have 3 connections to the server?

367 ?? S 0:00.40 imapd: tibor.domain [10.15.20.70] tmoldovan user.tmoldovan
372 ?? S 0:00.95 imapd: tibor.domain [10.15.20.70] tmoldovan user.tmoldovan.Apple Mail
4740 ?? S 0:01.65 imapd: tibor.domain [10.15.20.70] tmoldovan user.tmoldovan.Drafts

With 280 users that's about 840 imap Mail.app connections.

Message was edited by: tibor.moldovan

Reply

Answer 8

jaydisc

Level 4

1,464 points

Oct 26, 2008 1:50 AM in response to tibor.moldovan

Yes, it's quite common for multiple connections.

But, the test I was referring to was STOPPING the mail server and THEN see if that command yields any results, as if it does, that is a dead/hung process.

Reply

Answer 9

pterobyte

Level 6

11,104 points

Oct 26, 2008 1:54 AM in response to jaydisc

Missed the stopping part. Sorry.

Reply

Answer 10

Oct 31, 2008 11:17 AM in response to jaydisc

Thanks for clarification.
I haven't noticed any hung processes yet.

Reply