2 cards show up as My Card using a CardDAV on OS X Server

When my boss accepted a calendar invitation via email the recipient asked if my boss's colleague was replying to the invitation. After som digging, I found out that in his address book my colleague is listed as My Card. Checking my computer, she is listed there as well, along with some other computers in the organization. We use the CardDAV feature of OS X 10.9.5 Server 3.2.2.

When I select the right card representing him on his computer, and pick "Make this my card" the little icon briefly disappears from her name, only to return a second later. This happens on my computer as well.

On our second server, I had a look at the address book but there another user was "double selected". This was a former employee so I deleted him. A second later it selected not only the correct card for the server account, but also my female colleague's. Selecting some random card and making that the "my card" just picks another contact as a double.

When looking in the console, I get this message:

2016-05-03 14:35:58,149 Mail[6515]: -[MCContactsManager _processUpdatesForContactGroup:]: Count's pick a designated name contact out of 2 for: <CNContact: 0x7feefa4baa90: identifier=<CNContactIdentifier: 0x600000008420: stringValue=06E37F6F-E84B-431B-9565-96454F0D4B1D:ABPerson>, firstName=xxxxxx, lastName=xxxxx, companyName=xxxxxx, phoneNumbers=(

"<CNLabeledValue: 0x60000043c480: identifier=A88002E9-6EFC-4C49-8C9C-B0F4984EC819, label=_$!<Work>!$_, value=+46708519851>"

), emailAddresses=(

"<CNLabeledValue: 0x60000043c5e0: identifier=B353C61C-EE23-40AB-87E4-1526463F2BFA, label=_$!<Work>!$_, value=xxxxx@xxxxxxx>"

), postalAddresses=(

)>. Using the first contact out of the group