Error in downloading mail on IMAP after ios 18 upgrade

@ferrerod:

Here's the main part of the response I got from Fastmail who diagnosed and solved the problem on my account. It seems there was a bug in their mail engine that was benign until the iOS18 update, then no more. I hope this helps.

We had a bug which put random data into the HIGHESTMODSEQ value for some users. Their account was affected by this. We didn't think it was an issue because there was still heaps of number-space left, but the HIGHESTMODSEQ values were higher than a signed int64 can hold. This was fine because we use an unsigned int64_t internally, but the spec was updated a few years ago to make it a signed value, so we were strictly wrong.

Apple's parser couldn't handle these oversize numbers, and failed.

--

This has now been fixed by resetting the HIGHESTMODSEQ value for these users which also means this will never happen for them again.

ferrerod wrote:

Did fastmail tell you what technically they did? No one else has a workable solution. Some of us support our own mail serves - though perhaps rare.... would be great if you could get them to tell you exactly what they did for IMAP solution.

Addendum: PiHole on the local network caused the problem, because it blocks mask.icloud.com per default. This domain seems to be used for Apple Private Relay and anti-tracking services even if it isn't enabled in the iOS settings. Whitelisting mask.icloud.com in PiHole solved the problem for me.

My fix was to go to IMAP server name a type over the greyed out forward slash following the server name.

I backed out of settings and shut down the iPhone, and then restarted it a minute later. After the phone rebooted I went to mail and pulled down the mail to refresh it from the server. After a few seconds (which I think was the IMAP server resyncing with my mailbox, all my mail for the last few days came down. After two days it is behaving as it should with new mail appearing when it should. I note that this problem was not evident in my iPad which is iOS 17.x and not on my M2 MacBook Pro, nor on my partner’s iPad with iOS 18.1. I can only conclude that Apple can solve the mystery of this and the other issues with email that can’t be solved with this fix.

Looking at this thread it's fairly obvious that there are several different issues going on here. If deleting and recreating your mail account settings are fixing this, you had a configuration error. If using or not using a VPN is fixing this, you have a networking issue. If you're having a problem with a POP account, that's something else entirely. If the problem is intermittent, it's something else again.

In my case, and I believe many others here, the issue is a bug in Mail that was introduced in iOS 18 dealing with the IMAP folder separator character.

During the initial IMAP handshake, the server tells the client which character is used to separate paths for nested mailbox folders. e.g. If you have a mailbox 'Bills' with a mailbox 'Electric' in it, the path to that folder may be '/Bills/Electric', or '\Bills\Electric', or '.Bills.Electric', using whatever separator character the server prefers. There is no standard character used, but those three are the most common.

The caveat is that whatever character is used as a folder separator, can't exist in a mailbox name, so using '\' is fairly common as it frees '/' and '.' for use in mailbox names. Unfortunately, backslashes are also commonly used as escapes, and I suspect the IMAP code in Mail had a regression where a '\' character is getting misinterpreted as an escape during the IMAP handshake, causing an error and the client to drop the connection.

I'm running several Dovecot servers, some using '\' and some using '/'. iOS Mail fails on all the ones using '\', the ones using '/' work fine. Same version of Dovecot on the same server OS, all other Dovecot configuration is identical. When the client drops the connection, it's always immediately after the initial handshake, and there are no errors in the server logs. Even with debugging turned all the way up.

Those here running their own server, I'd like to hear what your folder separator is set to. If you change it to anything else, I expect your connection problems will go away.

Note that changing the folder separator has its own risks, Dovecot recommends you never change it. If any user has a mailbox name containing the new separator, their client (and possibly your server, depending on how the mail is stored on disk) will get confused (you'll see a single mailbox turn into one nested inside another). You also may break mail filters among other things. Make this change at your own risk. If you have the means, it's safer to stand up a new server with a different separator character, and then migrate the mail to it using an IMAP client.

Ideally, of course, Apple acknowledges and fixes the bug.

Interesting, thanks for that. It seems the we're either facing different issues, or there are other configuration settings that impact this. Do note that the excerpt you quoted above seems to be for shared mailboxes, you may have a different configuration for your inbox. Look for a namespace that contains "inbox = yes".

I just re-tried the experiment I tried a few weeks back and setup a test Dovecot server (v2.3.19.1 on Debian Bookworm). Setting the separator character to "\" (actually "\\" because it needs to be escaped in the Dovecot configuration file) causes my iOS mail connections to fail every time. Dovecot log shows a Login immediately followed by "Disconnected: Connection closed (LIST finished 0.009 secs ago)".

When I change the separator character to "/", ".", or "|" it connects and functions fine. Other clients (including Mail on MacOS) connect regardless of the separator character setting.

On my test server the only settings defined in the inbox namespace are "separator", and "inbox = yes" , everything else is the default value which should be equivalent to:

namespace inbox {
  type = private
  separator = "\\"
  prefix =
  location =
  inbox = yes
  hidden = no
  list = yes
  subscriptions = yes
}

Thanks to @cube_23 linking to a discussion/code change on the stalwartlabs github. Although not ideal, I am testing change on our dovecot server to eliminate advertising of the IDLE feature. So far in my limited testing things seem to be working, but it is too early to tell.

Does anyone else with this issue on dovecot want to test that change? You would need to change the 'imap_capability' configuration parameter in the dovecot conf files. In my case i used imapsync in verbose mode to get the current default advertised capabilities. I then removed 'IDLE' from the string and entered the rest into the configuration file. For example:

imap_capability = IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE XAPPLEPUSHSERVICE AUTH=CRAM-MD5 AUTH=PLAIN AUTH=LOGIN

then reboot dovecot and reboot the mail iOS client.

I was able to resolve the issue POSIX 96 by logging into Passwords and adding a new Password Profile using the same username and password. The one saved for the BT email seemed to conflict when using the original Password Profile in the Advanced Settings for the Mail

App. Once a new profile was set up I changed the autofill password and it resolved the issue and started to download the emails.

There are 2 error i have notice in my dovecot setup while my iPhone Mail app given POSIX 96 error.

First time dovecot given the error "imap-postlogin: Error: ERROR 145 (HY000) at line 1: Table './last_login/lastauth' is marked as crashed and should be repaired", that is have fixed by repairing that table in mysql.

Second time dovecot given the error "imap(......@..........in): Error: mmap() failed with file /var/vmail/vmail1/..../Maildir/dovecot.index.cache: Cannot allocate memory". For this error i have increased default_vsz_limit from 256M to 512M.

Overall My Understanding is Dovecot having any error in it plugin apart from IMAP protocol, other clients are ignoring it, where us the latest Mail app throwing the error in client side. So blaming iOS18 or new Mail app is not good.

May be i can help you to fix if you share the truncated logs while accessing Mail app to your server.

After over two months of not being able to use Mail app while other apps worked perfectly, my provider made some updates to the servers in co-op with Apple support. Even though its clearly issue only for Apples Mail app, therefore bug on their side. It is however possible to fix it on the side of mail providers and everyone having the issue, should ask them since Apple apparently won't fix this.

Asking your mail/server provider to fix this with Apple support is the only solution you can do on your own, nothing like changing `/` symbols and a reinstalling apps/mail accs didn't work for me.

Everything works now except the notifications, for those I am still forced to use Airmail. I suppose its just too much to ask for on a premium product such as iPhone... (and yes I have notifications allowed for each account)

My issue was fixed by going to IMAP settings and restyling over the greyed out forward slash after the IMAP server address.

I backed out of settings and shutdown the iPhone, and the restarted it a minute later. After the phone rebooted I went to mail and pulled down the mail to refresh it from the server. After a few seconds (which I think was the IMAP server resyncing with my mailbox, all my mail for the last few days came down. After two days it is behaving as it should with new mail appearing when it should. I note that this problem was not evident in my iPad which is iOS 17.x and not on my M2 MacBook Pro, nor on my partner’s iPad with iOS 18.1. I can only conclude that Apple can solve the mystery of this and the other issues with email that can’t be solved with this fix.

My understanding is IMAP server (dovcot) returning some kind of message which Mail app returning as POSIX 96. in my case i have iRedMail setup which uses dovecot to provide IMAP and that dovecot having the facility to record the each login time to mysql database. In that i had issue in that table mysql not able to update lastlogin details. the error is return to IMAP client (Mail app) and Mail app showing that as error. usually all other IMAP client apps will ignore this type of IMAP protocol replies and continue to do service. But iOS18 Mail app stopping at these type of un-handled error messages in IMAP communication.

So my guess is, 1) Your dovecot server also having such issue and returning un-supported/un-handled error reply to IMAP client. if you are running your dovecot in debug mode, you can find the issue and fix it. Then your iOS18 Mail app will work.

2) Apple also can improve the IMAP protocol handling in their upcoming Mail app releases like if such type of ignorable error coming from IMAP server, simply ignore it by showing warning or do no action and do the mail fetch and do the essential operation.

I have plan to reproduce this error in my server maintenance time. if i am able to reproduce it, i will update about the specific protocol error that may help apple/others to fix the issue.

I did some more testing and I isolated the problem. On my Dovecot servers that iOS devices cannot connect to, the folder hierarchy separator is set to "\\" (an escaped backslash). On the one that I could still connect to the separator is set to ".". I created a test server (v2.3.17) and confirmed that iOS will not connect with a backslash, changing this to pretty much anything else (".", "/", "|") allows iOS to connect again, set it back to a backslash and it fails.

Unfortunately it's not trivial to change the separator for existing Dovecot installations, but that's apparently what's causing the Mail app to choke.

Using a backslash is not uncommon and is what Outlook servers generally do, so this is not an unusual situation (I use it because it allows "." and "/" in folder names).

Hopefully Apple will fix this regression soon.

My iPhone was updated to 18.1 and I noticed immediately that my imap email account would not refresh for hours, despite sometimes advising all Mail had been downloaded. Whenever I checked the same account on my iPad or M2 MacBook the mail came down immediately. After the 18.2 update I thought issue might be fixed but after several days it is the same. Only iPhone is affected. The error message seen for the first time today is POSIX 96 - imap settings might be incorrect. All settings are the same on the iPad and MacBook and are correct on the iPhone. Conclusion - the update to 18.xx has caused the issues. I’ll try Spark instead of Apple Mail as I have been told that other Mail clients are not affected with this very inconvenient bug.

nxnw wrote:

I'm game to try, but have no idea how to do it. Google finds several inquiries, many of which have super-useful responses like "why do you want to do that?", and a few may have instructions, but they are not comprehensible to me.

OK - figured it out. I found the "imap_capability" key in the "20-imap.conf" file. It was commented out, so after some experimentation I was able to edit it so that the capability advertised was exactly the same as before, but without IDLE. I don't want to say exactly because I am still not super confident that I dod it correctly. This is a MacOS server IMAP installation.

Mail immediately started downloading on my phone.

I have noticed a negative effect, however. My macs evidently use map idle and (until this change) received email the instant it hit the server, Now it takes several minutes.

My iPhones started downloading e-mail once I logged off Norton's VPN