Time Machine Backup to Existing Encrypted APFS Drive

I have an external drive that is formatted as APFS (Case-sensitive, Encrypted) that I currently use to back up photos and store some media. I would like to enable time machine to back-up to this volume. If I try to enable time machine with "Encrypt Backups" checked, it tells me it needs to erase the device in order to enable encryption. If I uncheck "Encrypt Backups" it tells me it needs to erase the device in order to disable encryption. This feels to me like a bug in the time machine backup setup, because I can't see why an existing encrypted drive shouldn't be usable as a time machine backup drive as long as it is properly formatted. Is there any way around this or will I need to erase the drive to use it as desired?