Of course, as I've mentioned, HDDs and SSDs are designed to cope with normal use and rapidly changing data used in caches. Yes they can still wear out, hence why there are enterprise grade disks used in servers - usually in RAID configurations - that can cope with higher than average usage, plus RAID provides some redundancy allowing failed disks to be hot-swap (ie, replaced while the server is still running) without the loss of any data or the need for restoring from backups.
But let's get back to consumer products. Virtual memory (or swap files) was designed because of (1) the RAM limitations of computers, due to cost and design; and (2) pre-emptive virtual concurrency used in operating systems that swaps out one process to replace it with another; why should RAM be reserved for a suspended process when another desparate needs it?
But this is a history lesson, times have changed. RAM is no longer expensive and computers are designed to use larger amounts, so, why do we need virtual memory? The answer - which I think you have confirmed empirically - is we don't! Provide a computer with enough memory and it won't have to swap pages between physical RAM and virtual memory.
The nature of multi-core multi-threaded CPUs have probably changed this as well. Of course virtual concurrency still exists, with processes being swapped out when they required or are waiting for something else, such as the hard disk or another peripheral or interface, to make way for others. But now we have real concurrency; dual core, quad core - even six core machines (should that be sex core?), together with hyper-threading and multiple CPUs, the number of simultaneous threads could be 12 or more, which means processes are swapped out less and real RAM is always in use, so swapping out pages of RAM to virtual memory should also occur less. Replacing HDs with SSDs, and USB 2.0/Firewire 800 with USB 3.0/Thunderbolt reduces delays and reason for swapping out threads is less. Of course there are still reasons why we do need to swap out processes, mainly because we don't have CPUs with 100 cores (yet). A modern OS may have 100 different processes, some with more than one thread, and so swapping processes will still be required, but swapping of their associated memory is probably not required, however this will depend on the actual processes or applications you try to use simultaneously. Nevertheless there are only so many apps you can actively use at once, and only so much RAM they can use in total, and, since these are finite, there will always be a finite about of RAM that will provide enough for the applications to operate without the use of virtual memory.
The other point is, despite the speed of SSDs/flash compared with HDDs, RAM will always be faster. So, as long as you have enough RAM and use the appropriate combination of apps for it, you will have a faster more reliable machine.