How do I completely remove Instagram from my iPhone after deleting the app?

in difficult cases like this, it is usually best to back up your phone to wherever you normally back it up, completely erase all content and settings, then from the hello screen set up the phone again using the backup to restore it as it was, but without the rogue icons, and considerably more spare space.

Back up iPhone – Apple Support (UK)

Erase iPhone – Apple Support (UK)

Set up your iPhone or iPad – Apple Support (UK)

if it was me then I would force restart the iPhone and try again

Force restart iPhone - Apple Support (IN)

It erases everything, but if you are using iCloud (any cloud) for Photos, contacts, calendars, Notes, then they come back on setting up the phone again, anything not in any Cloud is backed up.