You can also use the Terminal to delete the file using the rm command. Type rm and a space and then drag the file to the Terminal window. Hit return.
If you want a GUI solution, use Automator to create a Service. Set Service receives selected Files and Folders in Finder.
Drag in a Run Shell Script action from the Utilities library. Set it to pass input as arguments. Set the code to
for f in "$@"
do
rm "$f"
done
Save the service with a useful name. Then, you can right-click on the items in the trash and select your service from the Services submenu. It will delete the files. There is no safety net. If you want it to confirm the deletion in case you accidentally pick that item from Services menu, then add an Ask for Confirmation action before the Run Shell Script. Type a message in the first line and a description of what the buttons do. You can even make it look like Windows by just having a yes and no button. Or, you can change the name of the Ok button to Delete so you don't need a description of what the buttons do.