Read email showing as unread

Using an iMac 2011 with Mavericks 10.9.4. Been having issues with the Mail client leaving mail that has been read in the 'unread' smart box. The account is an IMAP, and I have done the empty the whole thing and start over routine, and that works. However I don't want to keep having to do this over and over, since some of my boxes are large.

Instead of a fix, is there a solution to this problem? Or is there something that I need to do on the other end of the IMAP, since the email is on a domain I own, not gmail or yahoo.

Thanks!