IMAP support broken in iOS 7?

Steve,

If that's the case, then it would seem to point to a timeout issue on the talkback from the IMAP ... especially since I saw this anomalous behavior at least once on the voicemail interface.

I'm wondering -- what is the elemental difference in semaphore between "delete the first message" and delete the 31st message (or any arbitrary non-first message)?"

something about the display fails to see the pointer correctly

If what you are saying is correct, then presumably, what we're seeing on the iphone is an "unclean database" ... the messages are only legacy nodes in the local database on the client device -- a misrepresentation of state on the server ... is that true?

Something that goes to support this notion is that the effect occurs for both delete AND "move to folder."

Alternatively, since it goes to the voicemail app as well (theoretically) - is there something inherently DIFFERENT between an UPGRADED database (e.g. "I upgraded my 4 to iOS7") vs. an installed database ("I'm not having problems with my new iPhone 5s running iOS7")? It's likely a pollution in the upgrade path if they've moved to a different database architecture or schema.

I would rule out IMAP server from this as the behaviour on the iPhone is the same when deleting the message and the phone is in flight mode.

I just did a little testing with my admin.

It REALLY doesn't seem to be the server side -- like you said.

What seems to be happening is that the internal database gets out of sync with the server, and then fails to manage the changes -- the primary issue seems to be that the internal database is NOT HONORING "MESSAGE DELETED" STATEMENTS FROM THE SERVER.

I tested against two variables for four different states.

The variables are: message appears on server, messages does not appear on server (serverYes, serverNo), and messages appears on iPhone, messages does not appear on iPhone (iPhoneYes, iPhoneNo).

[Caveat -- I am using Thunderbird (a known good client) to represent "server state"]

ServerYes/iPhoneYes

In this state, both clients see the message. When I select delete on the iPhone (after a refresh), that message deletes on the iPhone, does not reappear on the iPhone, but DOES NOT delete on the server. As with other users, this may only happen on the first attempted delete.

ServerNo/iPhoneYes

This state is VERY INTERESTING. When reviewing the IMAP logs on the server, we receive "segmentation faults" -- essentially errors due to a request to act on a memory location that is no longer present. This would indicate that the delete request IS REACHING the server from the iPhone. The reason for this seg fault in this state seems to be because the iPhone "sees" a message that is no longer present, and requests to delete it in appropriately.

ServerYes/iPhoneNo

In this state, the other client (Thunderbird) sees the message, but the iPhone does not ... I have not been able to do much testing on this state - but it is the result of a the iPhone recording its own deletion after a ServerYes/iPhoneYes

ServerNo/iPhoneNo

Well, this is the goal, isn't it? 😉

Ultimately, with those states in track -- it would seem that the client is sending a request to the server and ignoring the response, then getting out of sync with the server.

Apple needs to step up on this -- why are they so silent?

mgmead wrote:

presumably, what we're seeing on the iphone is an "unclean database"

Yes I agree with that assessment. Can't speak to voicemail though as I don't have that many of them. This issue has happened on both new and upgraded devices for our family.

Interesting it happens in flight mode too...that would seem to rule out a server issue, but others have said upgrading the server software fixed it. I wonder if an index starts out corrupt and resetting the server software forces it to reindex all the messages, or something like that?

As far as Apple being "silent", at least here, these are user forums and not support forums. At least one other in this thread posted they were working with support already.

Have the same problem. Messages deleted from either Outlook or Yahoo servers using the Windows imap mail client on Vista do not exit the iphones Inbox folder and instead are stuck there as a 'phantom' copy. Have tried numerous fixes over the last few days but no joy. This problem coincided with my upgrade to ios 7

Have been forced to conclude Apple have broken their imap client...

I have two iPad tablets both upgraded to ver 7.0.3. We have a private mDaemon mail server with SSL connectivity to it.

IMAP is broken on both iPads.

It sort of works - can receive emails sometimes painfully slowly, but sending is hopeless. It seems like there is either an internal cache for the sent mail, or the IMAP dialog times out and hangs for ages.

The first sending email of a batch of tests goes within a few minutes (like 4 min minimum), but others sent soon afterwards take upwards of 30min to deliver. During that time the 3G egg-timer circles continuously.

I spoke to Apple today and they started to get me to do some resets on the device. I hung up on them. Idiots.

Also this message keeps coming up on the screen:

Cannot Get Mail

The mail server "smtp.bla.bla.bla" is not responding. Verify that you have entered the correct account info in Mail settings.

Your particular problem seems to be unrelated to that/those described elsewhere in this thread.

First of all, I wonder why your *IMAP* (i e "incoming") mail server is named something like "smtp.bla.bla.bla".

That's normally indicative of an *outgoing* (that is, using the SMTP protocol) server, although it doesn't have to be true (it is only a name after all, although convention dictates if it's named as one protocol, that's the one you'd expect it to support).

Have you by chance gotten the incoming and outgoing mail servers mixed up in your settings?

In any case, the problem seems network related rather than having anything to do with the IMAP server/client/protocol.

Both the send and receive mail servers are one and the same in our case. We have just one IMAP server. But thank you for your concerns.

I sent 8 test emails from one iPad and the first one delivered in 2 minutes. The rest delivered at times out of order, over a period of nearly 2 hours.

I closely watched the logs on the mail server during this test, and there was no delay in delivering mail that arrived at the server.

I also tested the server with a Windows client and delivery was immediate.

I also tested the iPad with both 3G and wifi connections with the same results.

"IMAP support broken in iOS 7" - yes that is the topic.

>"IMAP support broken in iOS 7" - yes that is the topic

But you're asking about an outgoing email issue which is SMTP. IMAP is used for incoming mail only, for downloading new mail. Try port 587 instead of 25 for SMTP. Some ISPs block port 25 to force customers to use their outgoing mail server and curb spam. Or search for issues with outgoing/SMTP email delays.

yep fair enough - my main problem is smtp which as you point out is not IMAP even though the account is IMAP.

The error message "The mail server is not responding" is an IMAP problem, but may or may not be unrelated to the sending issue.

I have started this thread .. https://discussions.apple.com/thread/5685687

I have 3 imap accounts in mail and they work about a week after

"Cannot Get Mail

The mail server "x" is not responding. Verify that you have entered the correct account info in Mail settings." error occures.

Only thing that works is delete account and re-add it. It's very disturbing.

7.0.4 ios installed. Any ideas?

POTENTIAL SOLUTION FOR UW-IMAP!!!!

WARNING: THIS IS NOT FOR THE COMMON USER - SYSADMINS ONLY

(if you act on this as a non-sysadmin, you may lose all your emails)

I did some deep work with my crew to try and solve this annoying problem.

What it comes down to is that, while it's affecting IMAP, it's really about your mbox/INBOX spool. That file is out of sync with the iphone, so the two no longer track per-message status the same and you end up with these out of sync deletes being ignored.

What you want to do is force a reset of the status tracking on the server -- the way you do that is by clearing that hidden internal "Dummy Message" in the spool file ... but there are some dangers ... the biggest one being that you have to lock the file while editing it -- otherwise, SMTP may come in with new inbound email while you're editing and bad things could happen.

The spool file I'm discussing is the one USUALLY found in a place like /var/mail/<username>

This dummy message retains "state" for deleted, read, replied, etc.

WARNING: YOU MUST LOCK THE SPOOL BEFORE EDITING!!!! (your server may use .lock or file locking -- if you edit while Sendmail tries to write to the file, bad things will happen)

We deleted my "Dummy Message", which in my case was at the top of the spoolfile. We deleted all lines down to the first "From" of the following message.

Once we deleted the dummy message, I restarted my client and PRESTO - true and righteous deleting. Appropriate status tracking between my iphone and other clients -- the whole fix.

PLEASE READ THIS ENTIRE MESSAGE BEFORE TRYING THIS ON YOUR SERVER - WE COULD BE VERY WRONG - YOU ARE TAKING FULL RESPONSIBILITY FOR DOING THIS (BUT IT WORKED FOR US)

This was completely figured out by my (genius) Sr. Admin, Pat -- he actually KNEW Crispin ... so we're talking about old school genius here (dear Apple -- learn a better word).

So, here are Pat's notes on what to do for locking and what to delete:

Pat says about the dummy message:

Dummy message looks like this.

From MAILER-DAEMON Sat Feb 22 15:00:05 2014

Date: 22 Feb 2014 15:00:05 -0800

From: Mail System Internal Data <MAILER-DAEMON@meadgroup.com>

Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA

Message-ID: <1393110005@meadgroup.com>

X-IMAP: 1159886868 0000096080

Status: RO

This text is part of the internal format of your mail folder, and is not

a real message. It is created automatically by the mail system software.

If deleted, important folder data will be lost, and it will be re-created

with the data reset to initial values.

Pat says about locking:

If you are gonna edit it, you need to lock it. If dot locking is supported

by your mail delivery agent, create in /var/mail a file named

<user>.lock, after you verify none is already present. Do your edit

and save, then remove the <user>.lock. If procmail is the delivery

agent, you can use lockfile /var/mail/<user>.lock. It won't

set it unless none exists. Otherwise use touch(1) Read manpage

for lockfile(1) before doing anything.

Procmail uses multiple locking schemes, including paying attention

to lockfiles.

If dot locking is not supported by the mail delivery agent, you should

shut down the mail service (sendmail, postfix, or whatever) to stop

mail deliveries, do your edit, then restart the mail service. Also make

sure no clients are active for <user> in question, so you are the only

one/process writing to the file.

This is to avoid possible corruption of inboxes by mail delivery scribbling

on inbox while you are editing.ssh

Not only causing possible loss of 1 or more messages, it is a PITA to

clean up.

I say about the process in general:

1. If you can, turn off your SMTP server briefly
2. Turn off all clients for the user in question (iphone, laptop, desktop, etc.)
3. Lock the spool file with either a file named <user>.lock or following the terms of your operating system's locking format
4. Copy your individual mail spool file to a backup (please!)
5. Find the Dummy Message in your spool (usually at top, but search for "INTERNAL" if not)
6. Delete Dummy Message
7. Restart SMTP server
8. Test with clients
9. Test with iPhone
10. If successful - come HERE and confirm method

mgmead wrote:

I did some deep work with my crew to try and solve this annoying problem.

What it comes down to is that, while it's affecting IMAP, it's really about your mbox/INBOX spool. That file is out of sync with the iphone, so the two no longer track per-message status the same and you end up with these out of sync deletes being ignored.

Thanks for the excellet debugging of the problem.

Many of us are using their ISP's IMAP server and are unable to make any modifications on server side.

Based on the information you have, is this something that Apple is able to correct with a new iPhone/iPad software, i.e. resetting the status tracking on the iPhone/iPad and not on the IMAP server? And somehow recognizing that the IMAP server and iPhone/iPad are out-of-sync?

I hope Apple has resolved this issue in the shortly upcoming iOS 7.1. And I really wonder what is the root cause of this problem, as this occurs only with some IMAP servers.

My guess is that it has to do with the initial setup of IMAP folders on your ISP.

Most standards put your folders somewhere besides your home dir, but uw-IMAP points by default to your home directory when queried for a location.

This has been the situation for years and years - and all standardized clients allow for a designation of IMAP path to overcome the client trying to map your home dir.

My experience has been that when the iPhone app is set up, the sequence of events is such, that it seeks to map before you can change the path variable (which is set in the advanced settings).

This creates a corruption if there is anything in your home dir that is not a mail folder (eg binary file, text file, etc).

This supposition is supported by the fact that the IMAP server core dumps as it is given erroneous requests by the iPhone app to process invalid files. We've found numerous core files relating to setup time.

This results in a corrupt map during creation on the iPhone - which in turn disables STATUS tracking of messages in inbox (presumably because that process of mapping happens AFTER updating folders, perhaps).

So the iPhone doesn't track state correctly in var spool.

OR

The coders don't understand the convention for tracking status and get lost?

Ultimately - the best setup by apple would have three parts:

1 - it would require a mail path prior to attempting to map folders

2 - it would have a "resync" button that forces the client to query the inbox dummy message for state

3 - it would have a clear means of actually removing the client side database. In my experience, deleting the account doesn't necessarily do it. I would see badge values on my mail client after the account was deleted.

Another thing that would help troubleshooting is if they would stop and start the app whenever changes are made in settings. Often, you go to fix something in settings - only to find the app is still up and may have retained values.

Apple has never been good with email - sad but true.

From curiosity -- where did you capture this log exchange for the iphone?