I noticed the same thing yesterday. Every so often I save off the SMS/iMessage SQLite database and delete the messages from my iPhone/iPad.
Using iPhone Backup Extractor, I extracted the iTunes iPhone/iPad backup. In /Library/SMS/Attachments I had 160 MB of files (photos, videos, PDFs, vCards, etc.) I sent and received via iMessage. Some were nearly a year old and were deleted in the Messages app months ago. (I usually deleted attachments immediately after sending/receiving them, but now I know that is a waste of my time.) These remained after I deleted all my messages and rebooted and resynced several times.
I assume at some point these attachments get cleaned up if space is needed, but who knows. The only way I can think of to delete them is to SSH into the device and delete that folder.
Also, the sms.db does not appear to be compacted even after several reboots, so I am not sure what triggers a vacuum.