I figured out that my 'phantom sound (vibration on silent)' was indeed coming from mail. More specifically mail that's being filtered to a label (sub-folder).
Before iOS8, when I had mail delivered to my inbox, I would get a sound (vibration on silent), notification & a badge. This still occurs properly after upgrading to iOS8.
However before iOS8, when I had redirected mail to a label (filter setting in gmail), I would NOT get any sort of notification which is the intended effect. Unfortunately post IOS8 upgrade, I get the 'phantom sound (vibration on silent)' when this occurs. No badge, no notification but I get the phantom sound (vibration on silent). This to me is a bug. I don't know if it's a delay bug where the mail is getting redirected too slowly and the sound (vibration on silent) is triggered or if it's just a plain bug where it's being triggered incorrectly. Either way it's a bug to me.
I don't want to turn off sounds/notifications for mail because I still want to be notified when I have non-filtered mail coming to my inbox. I just don't want to be notified when it's filtered.
ALSO another way I confirmed it was coming from Mail was by setting a custom vibration to mail. Each phantom vibration I got, triggered that custom vibration.