The caching server needs to be able to establish outbound connections to various Apple servers via ports 80 (http) and 443 (https). It also needs to be able to receive push notifications; the ports needed for that are documented at <https://support.apple.com/en-us/HT203609>. To receive requests from clients and peers it usually listens on a port chosen at random, but you can force it to listen on a particular port by, e.g.,:
$ sudo serveradmin settings caching:Port = 12345
and then stopping and starting the caching service. It may take clients and peers an hour or so to react to this change.