I had the exact same problem. I purchased a new 2020 iMac last month. 10-core i9. The Apple reseller I purchased it from upgraded it to 128GB of OWC RAM before being delivered to me. It was tested (by them) prior to shipping to me.
After a week of easy usage, I started hitting the machine hard with video encoding, really getting the temperature up high. It began crashing, especially when encoding in Final Cut Pro and using Safari at the same time. I could duplicate the crash each and every time and it would crash several times per day. After numerous calls to Apple support, and a trip to the Genius Bar after everything failed, the kept the machine for a few days and could not duplicate the problem. All their diagnostics said the components were fine.
Running out of options, the only thing I could question was the OWC RAM used by the reseller. I decided to remove the 128GB RAM and install the original 8GB RAM that came with the machine to begin my own testing.
The machine behaved normally immediately after I used the original 8GB RAM. The new RAM was the problem. I then used different combinations to determine which RAM card was bad. They all behaved fine in pairs, but not the entire 128GB together.
I then dipped cotton q-tips in alcohol and wiped the RAM connectors on the card. Three of the cards were clean, one card had a yellow residue that wiped off, in addition to being rough on the Q-Tip. One card had a residue that OWC's own website stated can sometimes occur.
I reinserted all 128GB RAM cards, and my machine has been working fine for the one week I've had it. I suspect that reside on that one card was causing major problems.
Give that a try and see if it works for you.