How much system time drift is acceptable?

Well, historically speaking, Apple internal clocks have always been a bit inaccurate, but ±3 minutes seems excessive unless the machines have been running without synchronization for a couple of years. Why don't you set the time automatically from Apple's server? (there's an option for that in the date/time preference panel) That should keep them in line without extra effort.

This reeks of an ntp server configuration error, some sort of network error, or of a bogus ntp server value.

Launch Terminal.app and issue the command

ntpdc -c peers

To see if the ntp clients are synchronized with the ntp server(s). (And how far off.)

Non-zero values for delay, offset and disp mean that the ntp client is locked. Zero values mean it's not locked.

General ntp information, as well as links to ntp debugging information, are available via the ntp.org site and the www.eecis.udel.edu site.

And FWIW, I would avoid getting any sntp servers into the mix, as can happen with some time servers.

I am not certain what you have done; what you're reporting. That the time drifts and that you are using ARD has been stated earlier. That the solution does not scale? Which solution? Invoking ntpdate certainly doesn't scale, of course, and the ntpdc command I showed doesn't reset the clocks; it's a query to learn the status on each client.

Put another way, what I posted was not a solution. It was a check to see if you have NTP services running on the OS X systems, and whether the NTP clients on those OS X systems were locked onto and with reasonable offsets from the NTP server.

Based on your comment (and what I can infer from this sequence), you have an NTP client running, and NTP is locked to the time servers (with a reasonable offset from the NTP server(s), and non-zero settings in the ntpdc output), and you are (still) getting clock drift?

If you are (still) getting clock drift, then your NTP servers might well be suspect.

Are you providing your own time-base? What time servers are you using?

Are your systems getting different offsets and jitter from the NTP servers?

Well, without being able to offer a solution, I can tell you that the sng-ntp-ext.asi server appears to be the problem. If I'm reading it correctly (see http://www.eecis.udel.edu/~mills/ntp/html/ntpdc.html), the = at the beginning means that the server is not synchronized but is "being polled in client mode" (the * at the start of the time.asia.apple server is a synchronized connection), and the numbers are just ugly: a polling time that is something like 36 hours and might be code for infinity, and an offset error of 488 second or roughly 8 minutes. The question is why those machines are not connecting to the apple server like they should. is there anything relating to ntp showing up as an error in console?

P.s. - the online forum has a threaded mode which may be a little easier to read.

Two of the three NTP servers referenced (the non-Apple servers) are not locked, and are apparently returning skewed times.

There are two ways to specify the NTP time servers with OS X Server. One is via the Server Admin GUI (Server Admin > select server > Settings > Time and Date > Date and Time > set the NTP server, and the other (and less common, less desirable) uses the /etc/ntp.conf file.

With OS X client time-keeping, System Preferences > Time and Date > Date and Time selects the server, or the /etc/ntp.conf file.

Modifying the ntp.conf file directly may (will?) get overwritten.

It appears that the two non-Apple time servers "have issues". Switch to the Apple servers.

Some related reading.

How these server acquired what are probably misconfigured NTP settings is something you'll want to or need to investigate.

There are also DNS errors with those IP addresses, too; the reverse translations are coming back with DNS that probably isn't what would be intended with a server.