Cannot delete "Xcode Caches" under System Information > Storage Management

I am using a 2020 M1 Macbook Air on Monterey 12.3.1 (21E258).

When I go to:

About this Mac > Storage > Manage > Developer

I see Xcode Caches taking up 4.85 GB space.

When I click delete, a pop up shows saying "Are you sure you want to delete Xcode Caches? This item will be deleted immediately, freeing up 4.85GB of storage. You can't undo this action."

I click "Delete" and then nothing happens. The item still stays there and doesn't delete. I've tried deleting it multiple times. I've also tried restarting my Mac and it's still there.

How can I purge these files from my computer? I also tried going into Xcode and deleting all of the simulators but that didn't change the amount of storage being taken up.