Can you delete voice memos from iCloud Drive without deleting from your phone?

I work a job where it's essential to frequently use voice memos. I like to have recent voice memos on my phone just in case I need to use them again.

My phone frequently tells me that it's not backed up because I don't have enough space in iCloud, but I don't want to pay for more space because the only reason I ever use iCloud is if I need a backup of my phone. (I already use a third-party storage service for photos.) I tried deleting old voice memos but despite deleting nearly 400 memos, the amount of space it said my voice memos were taking up (2.1 gigabytes) didn't change.

Is there a way to delete all the backups of my voice memos from iCloud, but keep them stored on my phone just in case? I don't feel comfortable permanently deleting all my voice memos from everything but at this point that feels like the only way to clear out enough space for a backup.