Which model of mini is it? You are running 10.4.4, so it is an older machine - is it an a powerPC G4 or an early intel model?
How much RAM do you have? The early mini's I think only shipped with 512Mb RAM, although the 2006 first gen. intel could take up to 2Gb. If you look at your memory use you can get a sense of how much disc swapping is going on (look at page outs). If your machine is disk thrashing a lot because of limited memory, then adding more memory can improve performance (otherwise it is pointless adding RAM to a system that is not limited by available RAM).
Other than that you are pretty limited in what you can do - the cpu is what it is, as is the disk, motherboard and firmware. If your machine is old enough that it shipped with 10.4, then it is upwards of 7 years old, so really, the only way to gain substantial performance imporvements when using up-to-date software is to get a newer machine.