You can open System Preferences > Keyboard > Text. Press the + button in the lower left, and in the replace column, enter “:(” (without the quotes). Now, under the Edit menu, lauch the Special Characters panel, or press control + command + spacebar. In the upper left corner of the Special Characters toolbar is a gear drop down. Select Customize List… and under Symbols, select Emoji. Press Done, and you now have an Emoji category present in your Special Characters left panel.
Now, regarding the first paragraph, double-click the With column adjacent to the replace content you just added. Find the Emoji that you want to represent the “:(" sequence, and double-click the Emoji from the Special Characters palette. It will be inserted into the With field. Quit System Preferences.
Looks like this:
Now, in iMessage, when you type your frown character sequence, a small panel appears below “:(" with the new Emoji. Just tap the spacebar and it is inserted.