More Files in My Media Folder Than in ITunes Library

Hello!

So I frequently add/delete stuff from itunes (I review mixtapes, new music etc. often). And when I went to consolidate my files, I saw that I have 32,000 files in My ITunes Media -> Music Folder. In My ITunes Library I only have 27,700 songs currently. I was under the impression if you moved a song to the trash when deleting it, it would also delete it from the linked ITunes Media Folder.... Does anyone have any idea how this discrepancy arose, and more importantly how to fix it? If I need to recover a backup from an external drive I don't want to have 4,000+ extra tracks that I previously deleted!

Many Thanks

-JEM