It's not like you were going to sweep the entire Sahara desert out of the speakers! 😀 😀 😀 If the Mac is plugged in (turned off, of course) and the charger is plugged in a proper three-prong outlet, that means the case will be grounded thru the charger so there will be no buildup of anything, dust or static. If you want to be even more paranoid, you can get one of those antistatic thingies you put on your wrist and make sure you maintain contact with the case while you sweep away. Just make sure the bristles of the vacuum brush don't get too deep into the holes cause you could damage the speakers.
Using a can of compressed air is even more of a no-no, cause you will be pushing the dirt deep into the innards, which is worse.