Do I need to turn off both iPhones when moving the SIM card from one to the other?

It is not absolutely necessary to power-down the phone to remove/replace a SIM card, but it would add a level of "safety" when handling electronic equipment.

If you read the support article you arrived here from, you will note that Apple does not specifically tell you to power-down the phone before removing/switching SIM cards.