Well, that's a question I have been facing since 2010 and no concrete solution yet! Apple just does not care about this little details which in my mind create perfection.
The solution most of the community propose is that you force quit your mail app!
It is neither healthy for your mac nor the mail app. Doing this creates an error report and makes you quit processing emails and data in the background leaving you at incomplete synchronising with servers your email is based on!
APPLE! You should care for minor details!!!