Note, Macs will only run the installer disc that shipped with them, and newer retail release discs.
Getting Gray, OEM, Update, or DropIn installer discs will not work unless the disc was designated for that age of Mac.
Since the iBook G4 was October 22 2003 to May 16, 2006,
And 10.3 was released October 24, 2003, and 10.4.6 April 3, 2006, that leaves you with the following possibilities with an iBook G4:
October 22, 2003 - April 18 2004 iBook G4s could use the 10.3.2 retail release as 10.3.2 was released December 17, 2003
April 19 2004 to October 19, 2004 iBook G4s could use the 10.3.5 retail release as 10.3.5 was released August 9, 2004
The newest iBook G4 model could not use 10.4.3 or earlier retail releases.
10.4's retail release was April 29, 2005, limiting all iBooks released after July 26, 2005 to Tiger and Leopard only retail releases.
So given your limitation of wanting Panther, be aware that it won't work with certain newer iBook models.