Can I install 20 GB of Ram on my iMac 27 inch 2011

I just upgraded my iMac, 27 inch 2011 model from 4 GB to 16 GB. I installed two 8GB chips. Can I also install the remaining 2 2GB chips? Or, is it best to either have a total of 16 GB to 32 GB on this machine. The Apple personal told me it's not good to use too much Ram on my iMac because it would somehow cause problems on the system.

I am running OS X El Capitan.

Thanks for your help.