Profile Manager does not send APN(?)

Hi

I am managing small Apple infrastructure in company. As server I have setup Mac Mini with MacOS Server to deploy profiles and be able to remotely lock/wipe endpoints (MacBooks). Almost year after setting it up I realise while testing one thing that from the very beginning there is problem with APN. Now I'm stuck with not fully functional server which I can access only remotely because Mac Mini is placed in other company branch. Unfortunately I was in big rush when I was setting it up and wasn't able to test everything and while making tests on test infrastructure everything worked fine. For now endpoints are communicating with server only at first login after startup/reboot and after that in random manner (sometimes in few minutes, sometimes in few hours). From logs I have managed get one important (I guess) line:

[DMDevicemgrd sendPendingPushNotifications]: APNS hasn't been initialized yet, unable to send push notifications

I have tested connection to APNs servers with app "Push Diagnostics" but it shows that there is no problem with connecting to APNs servers, also test push is working fine. When I am turning on logging for APS I get nothing (Resolve issues with Profile Manager in macOS Server - Apple Support). There even isn't created /Library/Logs/apsd.log file. I had renew APNS cert but of course it did not help. I did not renew Code Sigining Cert but it is still valid. I tried to restart Profile Manager (turned it off and on) but this also did not help. I am lack of idea how to fix this problem other way than reinstall server and reconfigure all endpoints but like I wrote earlier server is placed in other company branch so it's not so easy. Maybe someone have any idea how to fix this?

Posted on Apr 4, 2018 3:21 AM

Reply

Similar questions

13 replies

Apr 5, 2018 8:41 AM in response to lukaszn

The "always established" connection is probably for receiving APNS notifications, but the actual sending of push notifications is done via an on-demand connection to that gateway.push.apple.com service. However, given that Profile Manager doesn't think that APNS has been initialized, it's not surprising that it's not attempting to actually send push notifications.


Can you try the following? One a Terminal window and run this command:


tail -n 0 -f /Library/Logs/ProfileManager/devicemgrd.log | grep APNS


Then in another Terminal window, run these commands:


sudo killall devicemgrd

cd /Applications/Server.app/Contents/ServerRoot/usr/share/devicemgr/backend/lib

sudo ./dm_tool activateAPNS


Paste the output from the first Terminal window here if the results don't lead you to an answer.

Apr 4, 2018 11:45 AM in response to lukaszn

You should search the /var/log/system.log file for lines containing "apspd". Also, you should make sure your server can reach Apple's push servers. This page should help: If you aren't getting Apple push notifications - Apple Support. Basically, make sure this command gives you a "handshake failure" error (35) and not a timeout or server not found error:


$ curl https://gateway.push.apple.com:2195
curl: (35) error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

Apr 6, 2018 2:13 AM in response to mscott_mdm

I got only this on first terminal:

root# tail -n 0 -f /Library/Logs/ProfileManager/devicemgrd.log | grep APNS

1:: [42833] [2018/04/06 10:03:25.511] Incoming request: activateAPNS

0:: [42833] [2018/04/06 10:03:27.609] APNS topic = com.apple.mgmt.XServer.[UUID]

I had removed UUID (or at least it looks like some kind of UUID) because I'm not sure if it's safe to publish it.

Unfortunately commands used in second terminal does not help. Result of last command was:

root# ./dm_tool activateAPNS

Sending command activateAPNS

Response: {"result"=>"ok"}

Still, when I try to use "Update Info" or even "Lock" command there is no extra network traffic to Apple's servers. Endpoints are still connecting to server "when they want".

Also I tired this:

root# ./dm_tool sendPushNotifications

Sending command sendPushNotifications

Response: {"result"=>"ok"}

Unfortunately this does nothing in my case.

Apr 9, 2018 9:38 AM in response to lukaszn

Ah, OK. So the original problem you reported does not appear to be the problem you are having now. At this point, Profile Manager is sending the push notifications, so the problem is either with apspd (you might look in /var/log/system.log for 'apspd' lines again), or some issue with clients not receiving the push notifications. Both of those are more difficult to diagnose because various factors on your network can also impact this. [The #1 cause for this that I've seen are web content filters that either outright block connections, or more insidiously, play Man-in-the-Middle on the TLS connection and break it. If you have any kind of content filter between either the server or your devices and the internet, I would strongly recommend that you whitelist Apple's entire IP address range (17.0.0.0/8)]


If none of these ideas help, I'm not sure that I have anything more to suggest. Good luck.

Apr 5, 2018 2:27 AM in response to mscott_mdm

Thanks for answer. Unfortunately this doesn't give me any new information. I grep through all /var/log/system.log(.N.gz) files and there is no even one line containing "apspd". There is only "applepm com.apple.apsd[NNN]: = mps_limit 0" after every "killall apsd" used to reload process to turn on/off logging.

Also as I wrote in my first post I test connection to Apple's servers with app "Push Diagnostics", but I checked if curl will work and I get handshake failure - error (35). Also I check on netstat and there is always established connection to one of Apple's server IP (17.0.0.0/8).

I started tcpdump on all Apple's IPs and it shows me that server is connecting from time to time to NTP server, but when I am requesting "Update Info" in Profile Manager there is no traffic to Apple's server. Also there is some extra traffic from time to time but it does not look like sending requests from Profile Manager (on port 443 or 5223 but without any "reaction" on my testing endpoint). It looks more like keepalive.

Apr 17, 2018 4:16 AM in response to mscott_mdm

Thanks for answer. Unfortunately this also does not help. In system.log I see only entry about kill 9 on apspd. After that I made two requests in PM. One for "Update Info", second for "Lock" but no entry for apspd for this two requests. Also I have checked if there is new log file for apspd but I did not see anything that would match this.

Apr 6, 2018 2:20 PM in response to lukaszn

So if it has an APNS topic, it means APNS is initialized, but from what you pasted, it seems you are no longer seeing the "APNS hasn't been initialized yet, unable to send push notifications" message, so now it seems there is some different problem.


Can you try this filter on devicemgrd.log?


grep -Ei 'push ?notification' /Library/Logs/ProfileManager/devicemgrd.log

Apr 9, 2018 2:35 AM in response to mscott_mdm

Lot of lines like these:

1:: [45094] [2018/04/09 11:13:31.041] Sending 4 Push Notifications:

1:: [45094] [2018/04/09 11:15:38.385] Sending 1 Push Notifications:

1:: [45094] [2018/04/09 11:16:09.045] Sending 1 Push Notifications:

1:: [45094] [2018/04/09 11:17:40.638] Sending 1 Push Notifications:

Sometimes:

1:: [42833] [2018/04/06 10:55:11.736] Incoming request: sendPushNotifications

Apr 10, 2018 4:27 AM in response to mscott_mdm

Still there is no even one entry in /var/log/system.log containing 'apspd'. Also I'm sure that there is no MITM mechanism in our network. Also problem exist when using smartphone as hotspot. Beside that, while connected to company network on endpoint I can see with netstat that there is established connection to Apple's server on port 5223 and according to docs this port is used to communicate with APNs.

Well, still I need to looking for solution, maybe there is anyone with any other idea what is the problem.

@mscott_mdm, thanks you tried to help me 😉

Apr 10, 2018 11:47 AM in response to lukaszn

I'm sorry I wasn't able to help you. I just want to reiterate that port 5223 is for the receiving of push notifications, not the sending of push notifications. Devices that receive push notifications open a persistent connection to Apple on port 5223 so that push notifications can be delivered through firewalls. (Because the client opened the connection, not Apple's servers.)


But when push notifications are sent, they're sent on port 2195 or port 443 (pretty sure Profile Manager uses port 2195), and that connection is not typically left open—it is only established for the time it takes to send the desired push notifications.


The documentation Apple provides here TCP and UDP ports used by Apple software products - Apple Support doesn't really make this very clear, but the developer documentation for implementing push notifications does. If you have access, maybe the information here will help: https://developer.apple.com/library/content/technotes/tn2265/_index.html

Apr 11, 2018 2:33 AM in response to mscott_mdm

I'm aware that port 5223 is used for the receiving of push notifications. But what I wrote prove that endpoints are able to connect to Apple's server at this port.

About port 2195, 2196 and 443, I ran tcpdump on MacOS and on router. I was looking for connections to network 17.0.0.0/8 and to ports 2195, 2196 (according to docs these are for sending push notifications from Profile Manager) and 443. When I requested in Profile Manager for "Update Info" for my testing MacBook there was not even one packet sent from MacOS to port 2195, 2196 or 443. So problem still exist somewhere in MacOS Server. If this would be a problem with network filtering then I would be able to "catch" at MacOS packet sent from MacOS Server to Apple's server.

Apr 11, 2018 11:27 AM in response to lukaszn

Agreed. It sounds very much like there's still a problem on the server.


I just found a note I had about enabling debug logging for the apspd process, which appears to be what sends push notifications for Server. Try this:


sudo defaults write /Library/Preferences/ApplePushServiceProvider/com.apple.apspd.plist EnableDetailedLogging -bool TRUE
sudo killall -9 apspd


Then create some new commands in PM and look for messages logged by apspd in system.log.

Apr 18, 2018 1:58 AM in response to lukaszn

Ok. Now the most confusing part. Trying to find some clues so I'm looking for any file with logs that may help. Meanwhile made request to "Update Info". Completed almost in no time. So I made second request for "Lock". Also completed almost in no time. I turn on tcpdump, made another request. There is traffic at port 2195 to Apple's IP. I didn't do anything more than looking for more information (no configuration changes etc.) since my last post.

Question is for how long it will work, because now it looks like everything works fine.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

Profile Manager does not send APN(?)

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.