Q: Can I upgrade my iMac

you can replace your mechanical HDD with an SSD

https://eshop.macsales.com/shop/ssd/owc/imac/2008

you can upgrade to a total of 6GB RAM

https://eshop.macsales.com/shop/memory/iMac/Intel_Core_2_Duo_PC2-6400

Although it is true that, with modern RAM modules, you can go to a total of 6GB RAM, the cost for the single 4GB module required to get there is US$93, roughly 3X the cost per GB of other RAM for that model. The only place I know of that has the module and guarantees it in Macs is OWC that Jimmy linked. So there is no price shopping to be had for this module.

It is possible that you have some software installed that could be contributing to slowness. Before spending that much money on a computer that probably won't support the next Mac OS version, let us review your software config for known problem apps and processes.

Please download and run this free utility:

http://etrecheck.com

It is secure and written by one of our most valued members to allow users to show details of their computer's configuration in Apple Support Communities without revealing any sensitive personal data. It reports potential issues for evaluation.

Run the program and click the "Share Report" button when it finishes. Choose "Copy to Clipboard" from the resulting menu. Then return here and paste the report into a response to your initial post. It can often show if any harmful files/programs are dragging down your performance or causing overheating.