First, try going to iCloud.com on your computer, or to the Find My iPhone app on your phone, and see if the device is still showing under Find My iPhone for the Apple/iCloud ID you use.
If it is, and you see the option to "Erase" the device again, you should do so. It will ask for your Apple/iCoud ID which you will need to supply in order to turn off the Activation Lock that is still on the old phone. If you do not see an option to erase the phone, then it probably was erased correctly, and you can safely delete it from the Find My iPhone account. That should remove it after a bit from all of the other places you see it.
You can also check under Settings, click on your name, and you should be able to click on the old iPhone in the list, and when it goes to the device detail screen, the bottom of the screen should show a Remove From Account option.
You can also go to your Apple ID account, and see if it is listed there and remove it there by also clicking on the device: https://appleid.apple.com/account/manage
Cheers,
GB