I have been seeing this very problem with .vncloc files created pre-yosemite and in yosemite. Now all of my mac's have names like "Mac Mini" and "Kathyrn's iMac" where there are characters that need URL encoding. So as I read through other people having the same problem I decided to figure out why some things work and others do not. So I found the existing .vncloc files I had created pre-yosemite and opened them up in vi to take a look.
There are two properties in the file one with name URL and the other with targetAddress and they are different.
<key>URL</key>
<string>vnc://Kathryn’s iMac._rfb._tcp.local</string>
<Dict>
...
<key>targetAddress</key>
<string>vnc://Kathryn%E2%80%99s%20iMac._rfb._tcp.local</string>
...
</Dict>
So this gets me thinking. I drop back to the command line and type
open "Kathryn’s iMac._rfb._tcp.local"
And it works. but
open "Kathryn's iMac.vncloc"
cannot find the host.
So I edit the .vncloc and start experimenting.
Once URL is set to a URL encoded string I can right click on the entry in the dock again and open it. BUT. Screen sharing edits the file and removes the encoding and puts it back to English readable and then the second time the .vncloc file no longer works. So I drop back to vi and edit again. This time I change permissions on the file on disk so it is not writable. chmod -w Kathryn\'s iMac.vncloc. Then the #&#@&!@ program leaves my URL string as I left it and it works every time. Of course now I cannot change settings - but at least it works.
So my guess is that there is a bug in Yosemite. Something like pre-yosemite used targetAddress and yosemite uses URL to and something still "repairs" the URL to english readable - and they are now incomparable. I would guess that anyne having this working has no need for URL encoding. And of course everything works from the command line just fine.
So thank you Apple for a marvelous new bug - still not fixed in 10.10.4 that I installed yesterday.
Moral of the story - don't put spaces and apostrophe's in your machine names even though Apple does this by default. And/or lobby Apple to fix this back to pre-yosemite behaviour that actually worked.