Latency with Drum plug-ins

I have been recording this track, and I seem to be getting a lot of latency with my Drum plug-in, I don't even have a large amount of tracks. I even tried to bring down my buffer size, and I'm still getting this major latency when I hit the drum pads. I've already recording some drum patterns, and I'm just trying to add additional drums, but again this **** latency. Is there anything I can do?