I'm trying to install the latest version of iTunes and the pop up tells me that I need to install Mac OS X 10.5. How do I do that?

You have to purchase a Leopard retail install disk from either Amazon or eBay. Apple no longer supply or support Leopard.

Assuming of course that your iMac can actually run Leopard:

Leopard system requirements:

http://support.apple.com/kb/HT3759

OS 10.5.x is the max OS for Macs with a PowerPC processor. If your iMac has an Intel processor you can upgrade to Snow Leopard OS 10.6.x and possibly Lion OS 10.7

Mac OS X 10.5 Leopard installation system requirements

http://support.apple.com/kb/TA24950

Leopard is no longer available at the Apple Store but may be available by calling Apple Phone Sales @ 1-800-MY-APPLE (1-800-692-7753).

If you can't obtain a retail install DVD from Apple, look on eBay or Google the installer part numbers to possibly find at an on-line store. Here's what to look for:

MB427Z/A Leopard 10.5.1 install DVD

MB576Z/A Leopard 10.5.4 install DVD

MB021Z/A Leopard 10.5.6 install DVD (single user)

MB022Z/A Leopard 10.5.6 install DVD (5-user family pack)

Installing Mac OS X 10.5 Leopard

http://support.apple.com/kb/HT1544

Mac OS X 10.5 Leopard Installation and Setup Guide

http://manuals.info.apple.com/en/leopard_install-setup.pdf

After you install the base 10.5, download & install the 10.5.8 combo update at http://support.apple.com/downloads/Mac_OS_X_10_5_8_Combo_Update

The DVD should look like this

Caution - Leopard does not support classic mode. So, if you currently open OS 9 apps in classic mode, you won't be able to do this if you upgrade to Leopard.

 Cheers, Tom 😉