Hi,
The AppleScript for this is still there.
It can be set in Messages Menu > Preference > General Section in the bottom drop down.
The Sound/alerts pane has gone in Mavericks and sound can now only be changed for "Messages Received".
Although I must admit to not trying it I have presumed the AppleScript would then only work for Text Invites.
Having said that the AppleScript should be able to run as a separate even like an App from the Dock.
The Drop Down in Messages has an Option to Show In Finder effectively.
9:47 pm Saturday; January 11, 2014
iMac 2.5Ghz 5i 2011 (Mavericks 10.9)
G4/1GhzDual MDD (Leopard 10.5.8)
MacBookPro 2Gb (Snow Leopard 10.6.8)
Mac OS X (10.6.8),
Couple of iPhones and an iPad