If both Gmail and Apple Mail are running ...
I believe this part of your question is irrelevant. For "both Gmail and Apple Mail to be running", it sounds like you're talking about running two separate, independent, but possibly conflicting apps on your Mac. If that's what you're doing, you should shut one of them down.
The normal series of events would be that incoming email would hit your gmail account (out there on the web) and your gmail filters would be applied at that time. Then, the gmail server would send the incoming message to your Apple Mail, which would apply its rules at that time. This assumes you have your gmail account set up in the Apple Mail app.
Gmail server side filters will see your email first. Apple's Mail app will only see the email AFTER it goes over the network to Google's servers to check if there is any new email.
Also Gmail filters will work on your email whether you access it via the web, via an iPhone/iPad/iPod Touch (or other portable device), or via your Mac, or even access it via another email client.
Server side rules always act first.