Improve speed of Imac Retina late 2014 - memory question

Hi,

I have an Imac Retina 5k (27", late 2014).

I want to improve the processing speed so I can work faster in Photoshop.

I currently have 8 GB memory installed across two of the four slots (2 slots empty).

It looks like I can upgrade memory to 32 GB.

If I max the memory out will this have a significant impact on processing speed?

Or would it be worth perhaps just upgrading to 24GB? i.e. 2 x 4GB, 2 X 8GB?

Any advice, as ever, much appreciated!

Thank you,

Ben