TLDR; the problem was from an EFI boot loader that I had installed.
Took this to the genius bar yesterday. We ended up doing a fresh install, got rid of my Windows partition and installed back on a clean install of Yosemite. The problem went away. So I took my computer back home and started reinstalling everything. I installed rEFInd (a custom EFI boot loader I use on my other machines) and the problem came back. Uninstalled and reinstalled a few times to be sure and it is the culprit.
One of the big benefits of the boot loader (at least to me) was to not have to hold the option(alt) key on startup. It would automatically allow me to pick which partition to boot from on each and every boot up. Now if I instead held the option key (as you would without the boot loader installed) and manually picked the startup drive this would also "fix" the graphics issue I had without having to uninstall the boot loader. However I ended up removing rEFInd (or rEFIt, I tried both) completely anyway and the issue is gone.
I have a 512GB SSD on this computer and saw that there are some possible issues for drives over 500GB, but for hard drive corruption. If I had to guess the combination of the SSD and the boot loader is the cause of the graphics issue. (as odd as that sounds).