Profile manager tasks stuck on "Pending"

Have you tried re-enrolling one of the Macs recently? Does that complete, or does that also get stuck with a pending "Enroll Device" command?

Have you checked that your push certificate is still valid? If your server has been operating for more than a year, it could have expired. The best way to check is to look in Keychain Access:

1. Open Keychain Access
2. Select the "System" keychain
3. Select "My Certificates"
4. If the View menu has an item that says "Show Expired Certificates" be sure to select that.
5. Look for the certificates that have names that start with "APSP:". Click the gray triangle next to each one until you find one that shows a key name of "com.apple.servermgrd.apns.mgmt".
6. If you don't find any certificate in step 5, or the certificate you found shows an expiration date in the past, you've found your problem. You would need to get a new APNS certificate and re-enroll every device you manage. Hopefully this isn't your problem.

I've never found any documentation explaining the mechanism that triggers the profile push mechanism. Empirically, pushing to a large number of devices at the same time seems to choke the process at some point. Restarting the client devices and then logging in and out a couple of times seems to clear the blockage, but is a bit of a nuisance.

If anyone has a better solution I'd like to know what it is too.

C.

I'm managing ~100 Macs, and they also seem never to check in except on reboot. (Not managing iOS, tvOS nor watchOS devices.) I very badly need to know how to induce my Macs to check in with the Profile Manager as a regular thing, and what might be preventing them from checking in.

I've checked that APNS is working. The Macs are properly enrolled and have a good Trust Profile installed. And changes to the configs get pushed to them fine *when they check in*, which can be a matter of weeks between reboots, and that doesn't really qualify as "managed."

Philip,

Some things to check:

1) Are there any messages from "apspd" in system.log? I know you said APNS is working, but messages from apspd would indicate otherwise. (A lack of messages probably means it is working.)

2) Look through the /Library/Logs/ProfileManager/devicemgrd.log file for the string "networkSettingsChanged". If the last such line you see says something is "apparently NOT reachable", you may have some network connectivity issues.

3) Can you select one of these Macs in the Devices list and start an "Update Info" command? Does it complete?

4) Do the tasks that are stuck pending have an "@" symbol in the Target name? (i.e., "Joe@Joe's Mac") If so, these represent "user channel" tasks and they can't be processed until that user logs back in on that specific Mac.

mscott,

In order:

1) No appearance of that word in system.log. (And Push Diagnostics from twocanoes always succeeds, from the server and from the client.)

2) That message appears occasionally but is always supplanted by "is apparently reachable" within a couple of minutes.

3) Doing this is guaranteed to remain Pending until the Mac reboots.

4) All of our configs are machine-level, applied to groups whose members are machines.

The apsd service is running on all the client Macs that I've checked, too. Visible in Activity Monitor, and in top, but top says it's sleeping, waiting for a call that never comes. —P

Re-enrolling gets stuck too. Our Service Desk techs know to check the Profiles prefs pane to make sure it populates whenever they have to re-image a Mac, and give it another restart or two until it does.

Certs are present and current. There was another thread on here indicating that even a non-expired server cert could be the issue, so I've renewed it twice, to no effect.

They do all say "This certificate was signed by an unknown authority," though. Inside the cert, the Issuer Name is "Apple Application Integration 2 Certification Authority," and in the System Roots there are four Apple Root certs but none with that name.

Do those APSP certs need to be explicitly trusted? or is the needed root cert missing, and if so how to go about getting it? This is starting to smell like the cause.

It is normal for them to say they are signed by an unknown authority. That's not a problem. It seems pretty clear that you're having some kind of APNS issue, but it is difficult to know what, specifically.

Can you try this command on your server:

curl -k https://gateway.push.apple.com:2195

It should respond with "curl: (35) SSL peer handshake failed, the server most likely requires a client certificate to connect", but if it times out or gives some other response, then there's a problem with the server communicating to the APNS server.

XXXXXXX:~ YYYYY$ curl -kvvv https://gateway.push.apple.com:2195

* Rebuilt URL to: https://gateway.push.apple.com:2195/

* Trying 17.188.137.28...

* TCP_NODELAY set

* Connected to gateway.push.apple.com (17.188.137.28) port 2195 (#0)

* Server aborted the SSL handshake

* Curl_http_done: called premature == 1

* Closing connection 0

curl: (35) Server aborted the SSL handshake

and yes there is a proxy server. I'll talk to the security architect...is it just this gateway.push.apple.com:2195 that should be whitelisted?

If the proxy server is interfering with this connection, that would definitely cause the problems you describe. Opening port 2195 to gateway.push.apple.com is all that is needed to allow Profile Manager to send push notifications. Because it's SSL-encrypted from both sides, a proxy server that plays man-in-the-middle will break it. Your proxy server can't try to inspect these connections, it has to let them pass completely unmolested.

Bingo. Not the proxy server, but the intelligent firewall, that has a rule denying traffic on non-standard ports. This is SSL so since it's not on 443 it's blocked. And the network guy says the server is trying really hard to get through on 2195, every two minutes. Change Advisory Board meets Friday; wish me luck!

curl: (35) Server aborted the SSL handshake

Well, I'm running out of ideas. One last possibility: is there a proxy server on your network?

Maybe try this same command with more verbosity:

curl -kvvv https://gateway.push.apple.com:2195

Cool beans! Glad I could help. 😉