There are 2 reasons for the "Your system has run out of application memory" dialog box.
A) Your boot disk has very low free storage, and macOS cannot create page/swap files to offload virtual memory contents to disk. This is generally not the case, but I mention because if you do have very low free storage, it might apply. Depending on how much virtual memory is being called for, anything under 50-100GB free may trigger the message.
B) A process (or set of processes) have asked macOS for excessive amounts of virtual memory address space. Virtual memory address space requires macOS to create Virtual Memory Page Tables in the kernel address space to keep track of the application virtual addresses given out. Generally, if there is a memory leak (process asks for a virtual address range, uses the addresses, forgets to give them back, asks for another virtual address range, uses the addresses, wash, rinse, repeat), eventually there are so many virtual memory page table entries trying to keep track of the virtual addresses, that macOS no longer has memory available for applications, and you get the "Your system has run out of application memory"
If you look at Applications -> Utilties -> Activity Monitor -> View (menu) -> All Processes -> Memory (tab), you can see what processes are using lots of memory.