i have os x 10.3.9 and when i go to do a software update my system says its up to date so what should i do
Your system IS up to date - you have the latest version of Mac OS 10.3. If you want something newer, purchase the disk for it. Amazon and eBay are good resources for that.
is there any way i can upgrade with out buying a disk?
i downloaded the file it was the full thing 7.2 GB but it was a .dmg and my mac would not open it any ideas?
File for what? As I already explained there is no way around purchasing the disk. If you found a download of the operating system, that is illegal and has the possibility of infecting your computer with malware.
Retrieving data ...