Hi,
I would term this as being messed up.
In iChat and previous versions of Messages there was a plist that listed all the permutations that would work as a Smiley.
The Smiley seen in the Choosing panel at the right hand end on the Text Entry field would show you the most popular for any Smiley and you could learn them.
In Messages 8 (Mavericks) the Smileys are supplied from the Emoji Font.
Although you can get to see these by the panel/drop down at the end of the text field you do not get the keystrokes involved.
This is because they only apply to Emoji and in some cases the pics/icons are two characters in size
However when you type a common Smiley keystroke combo such as colon, dash, close parentheses you do get to see what the app will substitute and have the option then of not displaying it.

9:08 pm Sunday; November 3, 2013
iMac 2.5Ghz 5i 2011 (Mavericks 10.9)
G4/1GhzDual MDD (Leopard 10.5.8)
MacBookPro 2Gb (Snow Leopard 10.6.8)
Mac OS X (10.6.8),
Couple of iPhones and an iPad