Well, I found one culprit that causes distnoted to balloon. I use emacs for development, and have found that if I restart emacs, distnoted shrinks to a reasonable size. So, something in emacs is causing distnoted to grow and not release memory.
A side note regarding emacs... if you use desktop to reload your buffers on startup, sometimes emacs will use up a lot of CPU while it tries to start. I don't know what it is doing, but if I kill the spinning process, remove the .emacs.desktop file and then start, it works. Kind of defeats the purpose of using desktop I guess, but it does work sometimes. This all started with Mavericks, so I guess there is something broken there.