The files will rebuild themselves when you start mail again. The reason for deleting them was to get rid of any corruption that may have occurred.
The files in the trash should not influence the application itself, but you have to quit Mail before removing the files. I'm not sure what happened in your case.
Rules in Mail can sometimes be a little glitchy sometimes. For an Out of Office Response, rules should work. However, usually the providers website has a vacation response settings there. For Apple email, go to iCloud.com website and log in there with your apple id. In Mail preferences there, you can set your vacation response there. Should be the same for others such as ATT, Gmail, etc. Since you are setting it directly at the server site, you would insure a better response since you will not be relying on a computer or another mail program with rules which have to download an unread message to implement them.
For mail programs, apple mail is a good basic in my opinion, but there are many others such as Outlook. If you need a more complex program, then you will just have to look around an find one that suits all of your needs.