iCloud validation of Message-ID headers is incorrect, blackholes valid messages
I'm the maintainer of a popular PHP email sending library (PHPMailer) used on millions of web sites (e.g. it's used by Wordpress). I have had reports of a problem with some messages sent from scripts being successfully accepted by iCloud, but never appearing in inbox or junk folders, and never being bounced, i.e. they are being blackholed by iCloud. I have been able to reproduce this problem very easily.
This is not reasonable behaviour. If there is a problem with a submitted message (e.g. message format, header validation, SPF failure, spam filtering, DMARC), it should either reject or defer immediately (appropriate for a problem that's verifiable locally), or send a bounce later on.
If I send to and from exactly the same addresses and same server but using Apple Mail, the message arrives successfully.
Here is a complete transcript of such a delivery:
2017-09-13 14:52:32 CLIENT -> SERVER: EHLO mail.example.com
2017-09-13 14:52:32 SERVER -> CLIENT: 250-pv33p00im-smtpin004.mac.com
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-EXPN
250-HELP
250-XLOOP DDA159EB6A1497DD5FA798D7E7066CFA
250-STARTTLS
250-ETRN
250-NO-SOLICITING
250 SIZE 28311552
2017-09-13 14:52:32 CLIENT -> SERVER: STARTTLS
2017-09-13 14:52:32 SERVER -> CLIENT: 220 2.5.0 Go ahead with TLS negotiation
2017-09-13 14:52:32 CLIENT -> SERVER: EHLO mail.example.comhttp://mail.synchromedia.co.uk/
2017-09-13 14:52:32 SERVER -> CLIENT: 250-pv33p00im-smtpin004.mac.com
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-EXPN
250-HELP
250-XLOOP DDA159EB6A1497DD5FA798D7E7066CFA
250-ETRN
250-NO-SOLICITING
250 SIZE 28311552
2017-09-13 14:52:32 CLIENT -> SERVER: MAIL FROM:<joe@example.commarcus@synchromedia.co.uk>
2017-09-13 14:52:33 SERVER -> CLIENT: 250 2.5.0 Address Ok
2017-09-13 14:52:33 CLIENT -> SERVER: RCPT TO:<user@icloud.com>
2017-09-13 14:52:33 SERVER -> CLIENT: 250 2.1.5 user@icloud.com Ok
2017-09-13 14:52:33 CLIENT -> SERVER: DATA
2017-09-13 14:52:34 SERVER -> CLIENT: 354 Enter mail, end with a single "."
2017-09-13 14:52:34 CLIENT -> SERVER: Date: Wed, 13 Sep 2017 15:52:31 +0100
2017-09-13 14:52:34 CLIENT -> SERVER: To: user@icloud.com
2017-09-13 14:52:34 CLIENT -> SERVER: From: Joe User <joe@example.commarcus@synchromedia.co.uk>
2017-09-13 14:52:34 CLIENT -> SERVER: Subject: Subject
2017-09-13 14:52:34 CLIENT -> SERVER: Message-ID: <8d4e57561b34a17232777bb3ef446900874c34b0f7d0ab0fe621db84b97e3dcc@mail.example.com8d4e57561b34a17232777bb3ef446900874c34b0f7d0ab0fe621db84b97e3dcc@mail.synchromed ia.co.uk>
2017-09-13 14:52:34 CLIENT -> SERVER: MIME-Version: 1.0
2017-09-13 14:52:34 CLIENT -> SERVER: Content-Type: text/plain; charset=us-ascii
2017-09-13 14:52:34 CLIENT -> SERVER:
2017-09-13 14:52:34 CLIENT -> SERVER: test
2017-09-13 14:52:34 CLIENT -> SERVER:
2017-09-13 14:52:34 CLIENT -> SERVER: .
2017-09-13 14:52:35 SERVER -> CLIENT: 250 2.5.0 Ok
2017-09-13 14:52:35 CLIENT -> SERVER: QUIT
2017-09-13 14:52:35 SERVER -> CLIENT: 221 2.3.0 Bye received. Goodbye.
Having happily accepted the message without error, it is never delivered.
This is the actual message sent by the script:
-----------
Date: Wed, 13 Sep 2017 15:52:31 +0100
To: user@icloud.com
From: Joe User <joe@example.commarcus@synchromedia.co.uk>
Subject: Subject
Message-ID: <8d4e57561b34a17232777bb3ef446900874c34b0f7d0ab0fe621db84b97e3dcc@mail.example.com8d4e57561b34a17232777bb3ef446900874c34b0f7d0ab0fe621db84b97e3dcc@mail.synchromed ia.co.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
test
-----------
This is the message sent by Apple Mail:
-----------
From: Joe User <joe@example.commarcus@synchromedia.co.uk>
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7bit
Subject: test
Message-id: <70309335-BB74-4B1F-95BA-7074819F6243@example.com70309335-BB74-4B1F-95BA-7074819F6243@synchromedia.co.uk>
Date: Wed, 13 Sep 2017 16:55:16 +0200
To: Joe User <user@icloud.com>
MIME-version: 1.0
test
-----------
The Apple Mail message has a number of RFC contraventions, but that apparently isn't a problem.
I did a bit more detective work. An obvious difference between the two messages is the length of the Message-ID value. The script uses SHA-256 for the left hand side (before the @), resulting in a longer 64-char identifier. I tried shortening it by 1 character - and iCloud delivered the message!
This header is entirely RFC compliant; the Message-ID header is defined in RFC2822 section 3.6.4, which does not impose a length limit on the left hand side. The semantics of the right hand side of the identifier are (by convention) usually considered to be the same as for domains - which does impose a 63-character limit on dot-separated name segments, however this is neither relevant nor applicable to the left hand side of the identifier. I suspect that the authors of this mail server misinterpreted this definition.
This is a bug in iCloud (or at least in the Oracle mail servers that you use), so please fix it, and do not black-hole messages for any reason.
Mac Pro, OS X El Capitan (10.11.2), 32G 5T