Unfortunately, El Capitan has come along and broken this solution:
1) You can no longer unload ssh-agent:
# launchctl unload /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist
/System/Library/LaunchAgents/org.openbsd.ssh-agent.plist: Operation not permitted while System Integrity Protection is engaged
2) You can't edit the org.openbsd.ssh-agent.plist even as root, also because of SIP, so no way to add a timeout value to the storing of the key.
3) You can't disable it from starting due to the above.
Only thing I've been able to do is set a cron job to run 'ssh-add -D' every few minutes so any key that gets added to ssh-agent gets removed a few minutes later so I'll have to password decrypt it again on next use.