Hi,
In iChat and Messages 7 (Mountain Lion) there was a .plist in the app that listed all the combinations that could be used for each Smiley.
Even here on the Apple Support Community forums you can type colon, dash, close parentheses and not see a Smiley :-) but will do if you leave out the dash.
When I type the colon, dash, close parentheses in Messages I and offered a view of the selected smiley or a X to dismsss and send as plain text.

What the .plist did was to then call the correct pic and display it. (Plain text was sent in either direction and the App changed it if you set "Show Smileys")
Later versions of iChat and then Messages actually sent the Smiley Pic after pulling it from within the app.
Those Smileys have gone as has the .plist and some other arrangement is supposed to substitute the Emoji ones.
Although there is no "Show Smileys" option in the View Menu any more there is a Substitutions listed in Edit menu which includes Emoji (I can't work out though if it make any difference).

8:45 pm Sunday; October 27, 2013
iMac 2.5Ghz 5i 2011 (Mountain Lion 10.8.4)
G4/1GhzDual MDD (Leopard 10.5.8)
MacBookPro 2Gb (Snow Leopard 10.6.8)
Mac OS X (10.6.8),
Couple of iPhones and an iPad