Contacts.app asks for CalDAV-password 2-3 times before it works

I find that when a Mac client including a Yosemite client connects to the VPN server it takes a little while before the Mac client learns to use the DNS server defined by the VPN server. I see this for example by connecting to the VPN server and then trying to access an internal only File Server via its host name. Initially this will fail while using the IP address works straight away.

Once it is 'learned' the VPN server defined DNS server and is then able to resolve the host name of the File Server then the host name also works.

It maybe something similar is happening when you try access the CardDav server, initially it fails to resolve it but then later can. You could try testing this by doing a fresh VPN connection and rather than trying the Contacts application instead use the Network Utility application to try looking up the host name.

For what its worth I use Kerio Connect as our Mail, Calendar and Contacts server and it can be accessed at least currently without needing to first make a VPN connection. Clients still need to authenticate to access it though. This could avoid your problem.