Unapproved caller : Security Agent may only be invoked by Apple Software

Since I Upgraded to Yosemite I encounter regularly a strong annoyance :

After running for a while when I try one of the following actions :

Open keychain item

Repair permissions

Open an encrypted email

Change preferences in the panel

Log in iMessages

Log in iTunes / download Apps

And a lot of other actions

On my iMac, Yosemite Pop Up with the following message : "Unapproved caller : Security Agent may only be invoked by Apple Software"

And I can't do anything, message quits often, I can't don any of the actions listed , I almost can't work anymore with my Mac.

The only thing I can do is to restart and then I am able to work normally for a while.

I was wondering if it is a Yosemite bug ? Or if it could be a kind of trojan ? Or if it is a bad install ?

If anyone get the answer and has a solution to repair this problem thank you in advance.