I struggled with this issue for weeks as I was trying to switch over to my new iPhone 13. What I ended up doing is much more tedious than it needed to be, but I think it's worked so far.
I ended up purchasing a month of iCloud storage and saving only what I needed onto the cloud (mostly photos, messages for me) but NOT via a Backup (the backup method wasn't working for me because every backup still thinks I had over 400GB of voice memos, so my backup was super bloated).
Then on the new phone, I'm syncing everything I need by opening the apps and getting the info back. It's basically a fresh install as I have to download each app individually again, and a slow sync of photos etc.
It's a pain that we have to pay extra money to transfer our phones when it's definitely an Apple issue, but having spent weeks trying to figure this out, it wasn't worth it anymore. I'd rather pay a small fee to do the transfer, then I'll cancel my iCloud subscription.
I hope this helps someone out there. And remember, don't bother with the iCloud backup if you do this. Just sync the bare basics you need to your iCloud and transfer that way.