10 Replies Latest reply: Jul 25, 2013 3:13 AM by petrahu
Rusty Ross Level 2 (175 points)

I have enabled the Adaptive Firewall in OS X Server (2.2) under Mountain Lion 10.8.2 as per Apple's instructions:

 

http://support.apple.com/kb/HT5519

 

However, I get back an error everytime I try to enable it:

 

# afctl -f

No ALTQ support in kernel

ALTQ related functions disabled

pf enabled

Token : 18446743524496027528

No ALTQ support in kernel

ALTQ related functions disabled

Jan 22 17:41:50 server.domainredacted.com afctl[17998] <Notice>: Cannot update the Event Monitor config

 

 

When I try to alter a setting:

 

 

sh-3.2# afctl -T 10

Jan 22 17:42:09 server.domainredacted.com afctl[18005] <Notice>: Cannot update the Event Monitor config

 

 

 

Or when I try to disable it:

 

sh-3.2# afctl -X

Jan 22 17:45:29 server.domainredacted.com afctl[18021] <Notice>: Cannot update the Event Monitor config

 

 

 

I thought perhaps that afctl was having trouble writing to AdaptiveFirewall.plist in /Applications/Server.app/Contents/ServerRoot/private/etc/emond.d/rules

 

sh-3.2# ls -l /Applications/Server.app/Contents/ServerRoot/private/etc/emond.d/rules

total 0

-rw-r--r--  1 root  wheel   3344 Jan 22 00:11 AdaptiveFirewall.plist

 

 

But even adding world write permissions to this file didn't help.

 

I also wondered if perhaps afctl was looking for AdaptiveFIrewall.plist in the wrong place:

 

sh-3.2# ls -l /etc/emond.d/rules/

total 0

-rw-r--r--  1 root  wheel   822 Jan 21 20:01 SampleRules.plist

-rw-r--r--  1 root  wheel  8964 Jan 21 20:01 Xsan.plist

 

But copying AdaptiveFirewall.plist here (or symbolic linking the file in this dir) didn't do the trick either.

 

Anyone have any idea why afctl keeps complaining that it  "Cannot update the Event Monitor config" in OS X Server 2.2 / Mountain Lion 10.8.2?

 

Rusty

  • Michael Lake Level 2 (190 points)

    Hi Rusty,

    Although I can't help out, I'm feeling some comfort knowing I'm not the only one having this issue; it would seem that Apple's afctl implementation is severly broken at this point. I'm also running 10.8.2, but Server.app is at 2.2.1.

     

    Unfortunately, the emond system is far outside of my knowledge base, so I'm just feeling around in the dark at this point. One strange thing I noticed was that there are 3 identical array values for additionalRulesPaths declared in /etc/emond.d/emond.plist, all of which point to the ServerRoot inside ServerApp. Is this redundancy necessary? Could it cause issues?

     

    I'll be monitoring this thread, so hopefully someone will find a solution.

     

     

    P.S.: Is that your writeup for configuring the VPN over on the macminicolo.net blog? Lots of good info there, thank you!

  • Rusty Ross Level 2 (175 points)

    Yes, I have observed this on several clean installs of OS X Server (2.2) on top of 10.8.2 and the symptoms are the same every time. So this seems like buggy behavior. I have not yet tested OS X Server 2.2.1, but it looks like you have done so, and the symptoms are the same.

     

    I will say that manually adding an IP to the blacklist does block traffic from that address.

     

    For example,

     

    sh-3.2# afctl -a 1.2.3.4 -t 10

     

    ...will in fact block traffic from 1.2.3.4 for (approximately) the next 10 minutes, and:

     

    sh-3.2# afctl -r 1.2.3.4


    ...will once again allow traffic from that IP.

     

    But the Cannot update the Event Monitor config warnings are still unsettling. In fact, while I haven't been able to confirm this yet in testing, but I suspect that the primary feature of the Adaptive Firewall isn't even working: to automatically block IPs that are reported by emond to have failed T auth attempts.

     

    That's all I've got on this at present. I'll likely do some more experimentation when I get a chance.

     

    And, yes! I did write that Mountain Lion VPN doc for the great folks at Macminicolo. I am glad it was helpful to you.

     

    Best,

    Rusty

  • Michael Lake Level 2 (190 points)

    Hi Rusty,

    Thanks for the quick reply. I can indeed verify that the the "adaptive" part of the Adaptive Firewall is not working on my server, as it's the reason I started digging into this in the first place. Just yesterday I noticed in my logs that a bot was trying to brute-force login as user "admin" (good thing I don't have one!):

     

    -----
    2/4/13 8:28:02.864 PM log[241]: auth: Error: od[getpwnam_ext](admin,<NAUGHTY-IP>): No record for user
    2/4/13 8:28:02.864 PM log[241]: auth: Error: od(admin,<NAUGHTY-IP>): verify plain: lookup failed for user: admin
    2/4/13 8:28:02.865 PM emond[204]: Host at <NAUGHTY-IP> will be blocked for at least 15 minutes
    2/4/13 8:28:06.987 PM log[241]: auth: Error: od[getpwnam_ext](admin,<NAUGHTY-IP>): No record for user
    2/4/13 8:28:06.987 PM log[241]: auth: Error: od(admin,<NAUGHTY-IP>): verify plain: lookup failed for user: admin
    2/4/13 8:28:06.988 PM emond[204]: Host at <NAUGHTY-IP> will be blocked for at least 15 minutes
    2/4/13 8:28:07.052 PM emond[204]: Host at <NAUGHTY-IP> will be blocked for at least 15 minutes
    2/4/13 8:28:11.097 PM log[241]: auth: Error: od[getpwnam_ext](admin,<NAUGHTY-IP>): No record for user
    2/4/13 8:28:11.097 PM log[241]: auth: Error: od(admin,<NAUGHTY-IP>): verify plain: lookup failed for user: admin
    2/4/13 8:28:11.098 PM emond[204]: Host at <NAUGHTY-IP> will be blocked for at least 15 minutes
    2/4/13 8:28:11.157 PM emond[204]: Host at <NAUGHTY-IP> will be blocked for at least 15 minutes
    -----
    


     

    ...and it goes on like that for several hundred lines. I manually added the offending IP to my blacklist and the login attempts stopped immediately. This matches your experience, so at least the behaviors are consistent.

     

    In the logs above, emond appears to be triggering the block appropraitely, but afctl is either ignoring or not receiving the block. As far as I can tell, the problem appears to lie somewhere between emond and afctl, but I'm not sure where or how to troubleshoot that connection.

     

    My server is under AppleCare, so I will give them a call and try to file a bug report at the very least.

  • Michael Lake Level 2 (190 points)

    An additional (and confusing) update. The adaptive firewall may actually be doing something on my machine after all, but it's definitely not consistent. While digging through my logs again today, I noticed the events that I've pasted below. Apologies for the wall of text, but I've included the entire transcript of the attack for completeness.

     

    2/6/13 10:48:44.161 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:48:44.161 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:48:51.331 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:48:51.331 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:48:55.243 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:48:55.243 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:03.151 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:49:03.151 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:07.112 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:49:07.112 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:10.989 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:49:10.989 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:21.890 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:49:21.890 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:25.801 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:49:25.801 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:29.699 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:49:29.699 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:29.700 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:29.870 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:29.948 PM afctl[25763]: Address already in the blacklist, not added (timeout has been updated)
    2/6/13 10:49:29.954 PM emond[117]: 381912569.864889 Host at <IP-ADDRESS> was blocked for 15
    2/6/13 10:49:29.954 PM emond[117]: 381912569.864889 Host at <IP-ADDRESS> was blocked for 15
    2/6/13 10:49:29.954 PM emond[117]: 381912569.864889 Host at <IP-ADDRESS> was blocked for 15
    2/6/13 10:49:33.591 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:49:33.591 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:33.592 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:33.669 PM afctl[25764]: Address already in the blacklist, not added (timeout has been updated)
    2/6/13 10:49:33.675 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:33.754 PM afctl[25765]: Address already in the blacklist, not added (timeout has been updated)
    2/6/13 10:49:33.759 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:33.836 PM afctl[25766]: Address already in the blacklist, not added (timeout has been updated)
    2/6/13 10:49:37.477 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:49:37.477 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:37.478 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:37.552 PM afctl[25768]: Address already in the blacklist, not added (timeout has been updated)
    2/6/13 10:49:37.558 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:37.633 PM afctl[25769]: Address already in the blacklist, not added (timeout has been updated)
    2/6/13 10:49:37.638 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:37.720 PM afctl[25770]: Address already in the blacklist, not added (timeout has been updated)
    2/6/13 10:49:41.433 PM log[7449]: auth: Error: od[getpwnam_ext](server,<IP-ADDRESS>): No record for user
    2/6/13 10:49:41.433 PM log[7449]: auth: Error: od(server,<IP-ADDRESS>): verify plain: lookup failed for user: server
    2/6/13 10:49:41.434 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:41.511 PM afctl[25771]: Address already in the blacklist, not added (timeout has been updated)
    2/6/13 10:49:41.516 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:41.597 PM afctl[25772]: Address already in the blacklist, not added (timeout has been updated)
    2/6/13 10:49:41.602 PM emond[117]: Host at <IP-ADDRESS> will be blocked for at least 15 minutes
    2/6/13 10:49:41.678 PM afctl[25773]: Address already in the blacklist, not added (timeout has been updated)
    

     

    Note that I've made no changes to this server since my last post, but afctl does indeed appear to be working in those logs; there are no more login attempts from that IP after this excerpt. Additionally, I thought this might be a good sign for progress on getting Apple's KB Article (http://support.apple.com/kb/HT5519) to work, but I still receive the <Notice>: Cannot update the Event Monitor config error.

     

    Also, what's up with the auth errors logging after the IP's already been blocked? And there's a 4-second window preceeding them each time.

     

    Curiouser and couriser.

  • Rusty Ross Level 2 (175 points)

    Also, what's up with the auth errors logging after the IP's already been blocked? And there's a 4-second window preceeding them each time.

     

    Exactly. If the IP was blacklisted, and things were working as expected, it is my understanding that you would not see those auth attempts in the logs.

  • catkfr Level 1 (0 points)

    Thanks for sharing.

    I'm in the same boat... ML 10.8.2 with server 2.2.1. I'm having ssh brute force attempts. I've tried to install denyhosts (which doesn't work in ML since TCL wrappers are missing) and am now trying to get sshguard working from macports.

    sshguard seems to be doing something, adding stuff to pf firewall, but attack continues as if nothing happens. So I'm looking to make sure that the firewall is up and running as planned. When following the Apple instructions (http://support.apple.com/kb/HT5519), I get the following:

    $ sudo /Applications/Server.app/Contents/ServerRoot/usr/libexec/afctl -f

    afctl[4803] <Notice>: Cannot update the Event Monitor config

  • UptimeJeff Level 4 (3,445 points)

    @catkfr

     

    The AF simply creates rules which PF (packet filter) uses.

    Did you enable PF? You can find several articles on the topic.

     

    If you want to know if its all working, try consecutive bad logins and see what happens..

    Worst case, you get disconnected for xx (15 by default) minutes.

  • Rusty Ross Level 2 (175 points)

    I maintain that this is buggy even in 10.8.3 (Server 2.2.1):

     

     

    Here is one example:

     

    sh-3.2# afctl -T 10

    May 14 18:13:49 testserver.local afctl[3132] <Notice>: Cannot update the Event Monitor config

  • petrahu Level 1 (0 points)

    I changed these lines to /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.a pple.afctl.plist to block IPs after 10 failed login attempts:

     

            <key>ProgramArguments</key>

            <array>

                    <string>/Applications/Server.app/Contents/ServerRoot/usr/libexec/afctl</string>

                    <string>-T</string>

                    <string>10</string>

            </array>

     

    Therefore the Application Firewall doesn't work correctly, blocked IPs are never removed. I sent a bugreport to Apple, and now I have a "solution". But, yes, the Application Firewall is very buggy and unusable, at least for ssh. I'm using sshguard from Homebrew.

    The bugreports 12117251 (Dictionary attacks) and this one are open yet.

     

    Here a some notes:

     

    To get afctl logging work I changed /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.a pple.afctl.plist like:

     

        <key>ProgramArguments</key>

        <array>

            <string>/Applications/Server.app/Contents/ServerRoot/usr/libexec/afctl</string>

            <string>-v</string>

            <string>1</string>      

        </array>

     

    I added in /etc/asl.plist:

    ? [= Facility SBS_Security] file /var/log/system.log

    and HUPed syslog.

     

    To block the threshold of failed login attempts change in /Applications/Server.app/Contents/ServerRoot/private/etc/emond.d/rules/Adaptive Firewall.plist:

     

                          <key>hostBlockThreshold</key>

                            <integer>10</integer>

     

    Restart emond with

    launchctl unload -w /System//Library/LaunchDaemons/com.apple.emond.plist

    rm /etc/emond.d/state/state

    touch /etc/emond.d/state/state

    launchctl load -w /System//Library/LaunchDaemons/com.apple.emond.plist

     

    Restart afctl with

    serverctl disable service=com.apple.afctl

    killall com.apple.serverd

    serverctl enable service=com.apple.afctl

     

    The commands

            rm /etc/emond.d/state/state

            touch /etc/emond.d/state/state

    clear the state of emond, they shouldn't be necessary always.

  • petrahu Level 1 (0 points)

    I wrote:

     

    I added in /etc/asl.plist:

    ? [= Facility SBS_Security] file /var/log/system.log

    and HUPed syslog

     

    This is false! My error. Sorry.