Hi, andre -
Max RAM for that machine depends on whether it is a Rev. A or Rev. B model -
Rev. A - - 256MB
Rev. B - - 512MB
Unfortunately I do not know how to tell the difference between the revisions - perhaps someone else will post in with that.
Max hard drive = 128GB. This is a hardware limitation of the built-in ATA controller. Since iMacs do not have PCI slots, there is no way to add a better ATA controller.
Related to that, there is a hardware limitation on the early iMacs such as yours - all OS's must be placed within the first 8GB of the drive. This means any drive 8GB or larger must be partitioned such that the first partition is a bit less than 8GB - 7.7GB is a reasonable limit - and all OS's installed onto that partition.
Article #25249 - Gray or White Screen After Hard Drive Upgrade
As far as OS's go, it can use any version of OS 9, and any version of OSX up through OSX 10.3 (Panther). It can not use OSX 10.4 (Tiger) or later.
Important - before installing OSX or even booting to it, make sure the firmware on it is current, per those available for downloading from this Apple KBase article -
Article #86117 - Firmware Updates