There are 2 reasons for the “Your system has run out of application memory” dialog box.
A. Your boot disk has very low free storage, and macOS cannot create page/swap files to offload virtual memory contents to disk. Depending on how much virtual memory is being called for, anything under 50-100GB of free storage may trigger the message.
Apple menu (upper left corner) -> About This Mac -> More Info... -> Storage (scroll down a bit) -> Storage Settings...
B. A process (or set of processes) has asked macOS for excessive amounts of virtual memory address space. In order to keep track of the virtual memory address space, the kernel creates virtual memory page tables. If there is a memory leak (process asks for a virtual address range, forgets to give it back, asks for another range, forgets again, wash, rinse, repeat), eventually there are so many page tables created there is no memory left for applications, and you get the “Your system has run out of applications memory”.
Look at
Applications -> Utilities -> Activity Monitor -> View (menu) -> All Processes -> Memory (tab)
you can see what processes are using lots of memory. Many of these processes will be background agents and daemons used to provide many of the macOS services, as well as your applications.
Also keep in mind that each web browser tab will be a separate process running its own Javascript. If you have lots of browser tabs open, or if one of the browser tabs running Javascript with a bug in it, it is possible these browser tabs will add up to a lot of virtual memory demands, but no individual tab will look all that big.
