you might have some issue with the .plist
if you open terminal and type:
sudo defaults read /Library/Preferences/com.apple.SoftwareUpdate
you should be able to see all of the various keys and values that this plist contains. the first value -- CatalogURL -- should be your server name.
start your research there. and, what happens, if you REMOVE your server and point instead to apple:
sudo defaults delete /Library/Preferences/com.apple.SoftwareUpdate CatalogURL
to put your server back type:
sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL "http://your-server.yourdomain:8088/index.sucatalog"