How do I remove voice memos from iCloud without deleting them from my device?

I hadn't meant to sync my voice memos with iCloud in the first place, but now that they're all there, it seems like unsyncing them will only prevent new memos from being uploaded, rather than removing the existing memos from iCloud. I don't want them taking up space in my iCloud account, and I'd much rather they just live on my device instead, but deleting the iCloud information would also delete them from my device. It's baffling to me that there isn't a simple fix to this.