Hi,
Attachments to messages are actually stored in the mailbox along with the message text. It is when an attachment must be opened with another application, such as a Word document with Word, that the attachment is also placed in the Mail Downloads folder. There it can be used and edited by the application used to open it, and from there can be returned as edited to the sender or to others.
There is no reason files must permanently stay, but if you edit or otherwise alter files that have been attached, then you must elect some place to keep them for your own use.
You will note that in the Mail Preferences/General there is also a selection regarding when to remove files from Downloads folder if not edited. Any file that is not edited need not remain in the Downloads folder since it already exists in the mailbox with the message it came attached to.
Image files, since they can be opened directly in the Mail window, by Mail, do not go into the Downloads folder, but if any were opened from the message, with say, Photoshop, then that image file would also be sent to the Downloads, temporarily.
Ernie