So I did end up going to the Genius Bar, and showed them all my data. They agreed it was odd, and sent it to The Depot to have the software reloaded and retested. They sent it back after doing so and said it tested fine. This time, they did tell me that it could take 7-10 days to "learn" my habits, so I should make sure to give it that much time. I gave it a full 14 days, but was having the same issue. I was about ready to bring it back and tell them I wanted a new watch, when I decided to try one more thing in desperation.
I blew up ALL the data. I had to kind of play around and figure out what order to do stuff in, but ultimately I think it's this: You unpair the watch from the phone first, then you restart the watch, and go to the watch settings and reset it so it gets reset as if it's new from the factory. It will say "are you sure you want to delete all of the data from the watch?" and I replied yes. I also deleted the watch app and the activity app from the phone, then shut down the phone and restarted it. After that I erased all content and settings on the phone...yep. Desperate! I did try to remove the watch from the phone without doing that, but it seemed like it kept the data in the phone and wouldn't accept the reset. What's really weird tho, is that I did a restore on the phone data from a back up, and it pulled the watch app back in, but not the data. Anyway after I FINALLY cleared out all the crap data, the watch has worked like a dream. My actual activity time has been spot on every single time from the very first day, down to the minute.
There may have been an easier way to do this, but at least it works. Too bad Apple couldn't get to this solution with me.