Out of Application Memory After Upgrade MacBook Pro M1 Max 64 Gb Ram

Some of the reasons could be:

1. A specific application that is using too much memory: If you are running a resource-intensive application or multiple applications simultaneously, it could lead to the "Out of Application Memory" error. Try closing some of the applications or restarting your Mac and see if the issue persists.
2. Corrupted system files: Sometimes, corrupted system files can cause memory-related issues on a Mac. You can try repairing your disk permissions or running a utility like Disk Utility to check and repair any disk errors.
3. Low disk space: If your hard drive is nearly full, it could affect your Mac's performance and cause memory-related errors. Try freeing up some disk space by deleting unwanted files or moving them to an external hard drive.

You may also want to consider running an EtreCheck report, and optionally post it here if you need assistance with interpreting its results.

Ref: Using EtreCheck to Troubleshoot Potential… - Apple Community

There are 2 reasons for the "Your system has run out of application memory" dialog box.

A) Your boot disk has very low free storage, and macOS cannot create page/swap files to offload virtual memory contents to disk. This is generally not the case, but I mention because if you do have very low free storage, it might apply. Depending on how much virtual memory is being called for, anything under 50-100GB of free storage may trigger the message.

Apple menu (upper left corner) -> About This Mac -> Storage (tab)

B) A process (or set of processes) have asked macOS for excessive amounts of virtual memory address space. Virtual memory address space requires macOS to create Virtual Memory Page Tables in non-pageable kernel address space to keep track of the application virtual addresses given out. Generally, if there is a memory leak (process asks for a virtual address range, uses the addresses, forgets to give them back, asks for another virtual address range, uses the addresses, forgets again, wash, rinse, repeat), eventually there are so many non-pageable virtual memory page table entries trying to keep track of the virtual addresses, that macOS no longer has memory available for applications, and you get the "Your system has run out of application memory"

If you look at

Applications -> Utilties -> Activity Monitor -> View (menu) -> All Processes -> Memory (tab)

you can see what processes are using lots of memory. Many of these processes will NOT be applications. Just background agents and daemons used to provide many of the macOS services, as well as 3rd party background processes doing whatever that 3rd party app thinks it should be doing.

Also keep in mind that each web browser tab will be a separate process running its own Javascript. If you have lots of browser tabs open, or if one of the browser tabs running Javascript with a bug in it, it is possible these browser tabs will add up to a lot of virtual memory demands, but no individual tab will look all that big.pll

Hi

Sometimes, it could be caused by dodgy app on the background.

Try to remember, whether a new apps has been installed on your system, or not, or terminal apps (like modules to Python, or Brew app, or maybe there was some 3rd party system extension that asked your permission under System Settings/Preferences -> Privacy & Security