Well, when you implement such a way consider that spammers often use fake sender e-mail addresses; so either the "returned" mail reaches someone innocent or is getting lost somewhere on the Internet.
In worst case, your rejected message returns back to the server. I guess that OS X mail service is intelligent enough to detect that the mail has been already returned once, otherwise it could happen that these two mail servers (your OS X server and the target one) sending back and forth such undeliverable messages which can dramatically increase your network traffic.
So finally I don't see much sense in your efforts. I made the experience that a well configured SpamAssassin (with Razor2, DCC, other network checks turned on, trained bayes) is highly effective.