It's farily possible 10.9 changed the way audio plays back. Frankly, even prior to 10.9 (which I don't use, BTW), I'd not try to record audio when it's muted off (I wouldn't expect muted audio to be recordable).
While I'm sorry for you, I find it more logical/expected that way.
You know, switching to a newer OS version requires to change some habits. That's why I'm still using 10.6, as I hate changing my habits, and I'd be forced to if I upgraded (I know, I tried shortly every releases up to 10.9). I'm afraid it's fairly possible you can't do that any longer. Why can't you just switch the speakers off?