Thank you everyone for your comments and help. We had to push the release back two weeks for many reasons, some of them this issue.
But I thought my solution would be good to know for people. For the record I was attempting to edit at 4k with imported RED files converted to optimized media. What I didn't realize is that some of those clips were actually shot at 5 or 6k. I think that was one problem in and of itself, so I converted my projects to 1080p. Still got memory errors though.
So I think what solved the issue was when I flushed the cache by moving it outside the library. I think FCP had crashed while doing background renders from choking on some of the odd clips. I believe that corrupted some of the cache system, and wasn't fixed even when regenerating the cache files.
So I went to project settings and set the project cache to be outside of my project file in a folder I created. When doing so you are prompted to decide whether or not to move your current cache. I didn't, and this forced it to re-reender all those clips (but not the original optimized or proxy media). This seemed to clear up every problem I was having with Final Cut Running Out of Memory problems. I bet it was the cache, but could have been related to the mixed resolutions.
So I thought this might helpful to others searching this forum, and thanks again for everyone's help!