I would get 2x32GB now, install them in place of the original 2X8GB and return them for a small rebate from OWC.
https://eshop.macsales.com/item/OWC/2666DDR4S64P/
Rebate info:
https://eshop.macsales.com/tech_center/support/rebates/
Check prices periodically for when they drop and get the other 2X32GB in a couple of years, if you ever think you need more and want to max out the machine. As faster computers and RAM are released in the future, the older RAM tends to drop in price. So, the stock RAM you have will be worth less as a rebate or for re-sale in the future as well.
RE: original 2X4GB??? Well, adding more will definitely improve the performance of your machine. That bare-minimum offering of RAM in a base model is really not enough, even by a few years ago standards. Go ahead and return it for the $10 rebate, having that extra 8GB will probably make very little difference once you have the 64GB RAM in there.