I am afraid I'll just add to the confusionby saying what I noticed earlier today.
As I said, my Software Update server is my own Mac Mini server, sitting right next to me on my shelf here. It's running Lion Server and has been working fine for several years. Even after upgrading my iMac to Mountain Lion (10.8.1), I noticed no problem with it checking the Mini for any new software. That is until this morning, when I read about the new iTunes and did a Software Update to install it.That's when the error popped up.
During this discussion, when I posted my earlier message here, I checked the server and the iTunes 10.7.1 has been downloaded and was ready for clients to install. However my iMac Software Update couldn't see it. Then I run Software Update in my Mac Mini server. Obviously that machine checks Apple's software update servers, and it did find iTunes new version and installed it fine. The difference between the two machines is that the Mini uses the traditional Software Update, while the iMac uses the new App Store update procedure.
In the past, on the iMac, I've been using a program called "Software Update Enabler", which allows you to enter the URL of the server (instead of going through the Terminal with the default write command). So I run it and sure enough, the settings were pointing to my Mac Mini server for Software Update. So I tried saving my settings again, and for the first time, the program was trying to verify the server but could not. I Forced Quit the program and run it again, this time resetting the software update server, to point to Apple's servers. Run the software update again and the App Store reported that there were no new updates. As I said in the previous paragraph, just seconds ago, the Mac Mini found new software and installed it. The iMac (via the App Store) couldn't find any updates (and yes, I've verified that the iTunes on the iMac was the old version).
At that point, I decided I had enough and downloaded iTunes from the Apple site, as a direct download.
However, the Software Update Enabler still can not set back the software update server to point to my Mac Mini server. So I used the Terminal command:
sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL http://su.myserver.com:8088/index.sucatalog
I verified it and it is set correctly. However, I am not sure at all if my iMac is currently checking my server or Apple's.
That issue is VERY strange.