Can I mute the incoming messages in the Messages app?

Try

System Preferences (the gear icon in your dock)

Notifications

Messages

and Uncheck Play 'Play Sound When receiving messages'

HI,

Two Ways (At least)

1) In Messages menu > Preference > Alerts (Different items in the top item drop down and set no sound )

2) System Preferences > Sound > Sound Effects and mute the effects (This will be all Alerts system wide but it the Volume Setting iChat and Messages uses at this point)

3) As Damien-007 says.

4) If you happen to use Speakers for some things and Headphones for others you can set the Output of Special effects in 2) to something else (this is really a variation on 2) )

9:45 PM Friday; August 3, 2012

Please, if posting Logs, do not post any Log info after the line "Binary Images for iChat"

 iMac 2.5Ghz 5i 2011 (Lion 10.7.2)
 G4/1GhzDual MDD (Leopard 10.5.8)
 MacBookPro 2Gb (Snow Leopard 10.6.8)
 Mac OS X (10.6.8),
"Limit the Logs to the Bits above Binary Images."  No, Seriously