Just to be certain it isn't truly installed (I realize you did state that you did NOT install it), check under Settings/General/Usage/ then under Storage, Manage Storage. Look for it in that list. If it's in there, tap on it and then tap on the Delete App button.
Is it in your purchases? Check in the App Store/Updates/Purchased and look for it in that list, just to verify that it was/was NOT purchased (you can't do anything more from here).
If this fails and it continues to show, sign out of your Apple ID under Settings/iTunes & App Store, tap on your Apple ID, then tap Sign Out.
Next, reset your device:
Device Reset (won't affect settings/data/music/apps/etc)
1. Press and hold (& continue to hold) BOTH the Sleep/Wake button & the Home button.
2. Continue to hold BOTH (ignoring any other messages that may show) until you see the Apple logo on the screen.
3. Release BOTH buttons when you see the Apple logo and allow the device to boot normally.
After the device has restarted, sign back in to the App Store as above.