How do I stop iTunes Match from uploading Voice Memos?

I understand that iTunes Match is NOT supposed to upload voice memos, but half of mine have been uploaded. This is a problem since many of my voice memo's contain confidential medical information. I will be removing all voice memo's from iTunes, but that doesn't solve the problem when I want to listen to them, since they will then re-load themselves into the iTunes library when I play them. Any feedback is appreciated.