I see that you've already gotten some very good advice here. In particular, I want to stress the importance of what Linc has said. What happened was a very serious crime, and needs to be reported. For all you know, the creep on the other end of the line was trying to get a webcam hack installed, not a keylogger, in hopes that he could spy on your son. Call the police, please!
Now, as to the software that was sent... Most likely, it really was a Windows-only exploit. However, we really don't know how deep a conversation your son had with this man, and it's entirely possible that it was actually a Mac backdoor that was sent. We also don't know what the message was about something needing to be downloaded... I don't see any reason for that message, as a malicious app or installer would be able to download stuff without asking, once it got its foot in the door (so to speak). Worst of all, it sounds like you have deleted the files, so those of us who could help you figure out what they might have been cannot do that.
Ultimately, your response to this is your decision. I have an extremely good relationship with my teenagers, and yet if the same thing happened here, I would not make any assumptions. I would consider the worst-case scenario - that some stranger might have gotten something installed that will let them spy on my child - and would respond to that.
In that case, there's only one reliable way to handle that worst-case scenario. Completely erase the hard drive. Then, if you have one, restore to a backup from before this incident happened. If you don't have one, reinstall the system and all applications from scratch, and copy personal documents only (no settings files, and no using Setup Assistant or Migration Assistant for the import) from a backup.