When one restores an iPhone from its own backup, visual voicemail is also restored. However, when one restores a new iPhone from a backup from a previous phone, the voicemails fail to restore. This occurred between my 3G to 4s, and again from the 4s to a 5s. Restoring the 4s once from its own backup resulted in voicemail recovery.
My only explanation is that when a different SIM card is used, voicemails cannot restore as one would otherwise expect. Perhaps voicemails are inexplicably tied to their SIMs; I believe this may be why your VMs vanished when you switched SIMs.
Unfortunately, newer models of iPhones tend to use exceedingly different sizes, rendering one possible fix (swapping SIMs) unpursuable [not your example, I know]. iBackupBot is the other possible solution: you can extract the voicemails from the backup and restore them onto the new iPhone individually.