Windows Server DNS implements a connection between DHCP requests and DNS services; check with the Microsoft Windows Server documentation for details of that. Here's how to implement Dynamic DNS (DDNS) on OS X Server. Whether VMware itself gets involved here (and it may), I don't know.
Follow the DHCP reservations around; that's the most likely path for this stuff.
SMB (a port of the Samba server, on this OS X release) is a Windows-specific file and disk storage service, and not particularly connected with DNS or DHCP operations.
So we have narrowed this down to the mDNSResponder service. When we stop the service the DNS records are unregistered. When we restart the service the DNS entries are created. We don't see any way in the MAN pages to tell the mDNSResponder service which adapters to register or exclude. Is there a way to exclude vmnet1 or vmnet8 from the adapters that get registered in Dynamic DNS?
So one of our techs found a solution that works for us. I am passing it on in case someone else runs into the same problem.
We were unable to find a method of configuring the mDNSResponder service, and we really even failed to find anyone who understood how that service works. On the VMWare side, there is a configuration file that tells VMWare fusion how to configure the virtual switches (VMNET1 and VMNET8). You can have the VMWare virtual switches provide DHCP and NAT translation even if you do not make the switches visible to ifconfig. the file to edit is located at the following location:
/Library/Application\ Support/VMware\ Fusion/networking
Edit the file and change the lines below in red from yes to no. We left DHCP and NAT set to yes, and that seems to allow the system to continue to operate normally even without the virtual adapters associated with the virtual switches.
answer VNET_1_DHCP yes
answer VNET_1_DHCP_CFG_HASH EE9700F0C9D80591CE6A3FD186CC21B6431CDA0A
answer VNET_1_HOSTONLY_NETMASK 255.255.255.0
answer VNET_1_HOSTONLY_SUBNET 192.168.217.0
answer VNET_1_VIRTUAL_ADAPTER no
answer VNET_8_DHCP yes
answer VNET_8_DHCP_CFG_HASH 3F1A8E60BC64C2E12ABE50AF9A9F2BAD75A973A4
answer VNET_8_HOSTONLY_NETMASK 255.255.255.0
answer VNET_8_HOSTONLY_SUBNET 172.16.200.0
answer VNET_8_NAT yes
answer VNET_8_VIRTUAL_ADAPTER no
I have the same problem and wanted to thank you for the helpful suggestion.
One caveat is that (at least for a NAT'd guest) I don't see any way to address the guest, so it can't run server processes. Applications on the host cannot route to the guest because they don't see a NIC on its subnet. So you cannot export a file share from the guest, or ssh into it, etc.
This worked for me (on Mountain Lion, anyway):
On (Mountain) Lion:
dsconfigad -restrictDDNS "en0, en1, en2, en3, en4, en5, en6, en7, en8, en9"
On (Snow) Leopard add the following lines to the end of /etc/smb.conf:
interfaces = en0 en1 en2 en3 en4 en5 en6 en7 en8 en9
bind interfaces only = yes