First, be absolutely sure it is a duplicate.
Check that Safari is still present in the Applications folder and opens when double-clicked.
This sort of thing usually occurs when dragging an application to the Dock and releasing the mouse too soon. In which case it would be the original on the Desktop.
As Safari is owned by the System and you don't have read/write privileges on it, it may require some Terminal intervention to achieve the deletion.
I'll leave that to someone who's more familiar with Terminal.
If you dragged the Safari icon from the Applications folder you will see a little arrow next to the desktop icon, and Safari is still in the Applications folder: drag the icon to the Trash it is just an alias (you do not need Terminal for this). If somehow the icon does not have a this little arrow drag it back into the Applications folder, and be sure not to delete it.