Yes, I think that's the best solution: run your iMac with 'only' the two 32GB 2666 MHz modules installed, in slots 1 and 3 (see the diagram that dialabrain posted above). With 64GB of RAM your 2020 iMac will be able to serve you very well.
No need to thank me. Helping others helps me to expand my knowledge too. I live in Europe and can't really come over for a beer or coffee, but appreciate the offer much! :-)
Let us know, if you want, if everything runs fine and at 2666MHz with the 2 x 32GB installed.
Dual Channel functionality? I will explain it in a simple way. Imagine your iMac as being a car with a big engine and lots of power. It will run better and faster if you allow it to use a lot of fuel, which we could compare to the amount of RAM you install. Instead of just installing a fuel tank of a 'normal' size (let's say your current 32GB of RAM), it's possible to install one that is twice as large (your new 64GB of RAM). That's already nice. However, in a non-Dual Channel world, all the 'fuel' needs to find its way to the engine by running through only one tube, one 'channel'. In your Dual Channel car/iMac, there are actually two fuel tanks installed, each with its own 'channel' through which the fuel reaches the engine (the processor). Result: the same amount of 'fuel' (the RAM or Random Access Memory that is 'consumed' or used by the operating system and the applications running) reaches your engine twice as fast in this 'Dual Channel' system as it would in a non-Dual Channel system.
This is also why Apple recommends, in case you install memory modules of different sizes, to minimise the difference in capacity between Channel A (slots 1 and 2) and Channel B (slots 3 and 4). In other words, it is best to have the two 'fuel tanks' as identical in size as possible.
I hope this answers your question about Dual Channel technology. It is a matter of allowing the Intel processor to access the RAM much faster.
I'm pretty sure that if you "only" install the two 32GB modules, one in slot 1 and one in slot 3, your iMac will be more than happy to run your applications very smoothly. That way you avoid having either the memory modules' speed lowered or the Dual Channel functionality disabled.