jeffzimmm

Q: OD performance grinds to a halt with 44 student logins - What do?

Mac Mini server, Late 2013 model, running 10.9.4.

 

Network is based on a Cisco rv042G router that connects to the Mac Mini, two 24-port Netgear gigabit switches, and a wireless AP. All Ethernet is Cat 5e.

 

Computer lab has 44 Macs with a mix of Ethernet + WiFi, iMacs, MacBooks, and MacBook Pros, all running either Lion or Mavericks.

 

With 15-20 students, things work OK. But today we had a session of 44 seniors who needed to start their Common Applications for college, and it was a disaster. Half could log in only after 15 minutes of waiting, the other half not at all. Nobody could reliably web-browse.

 

I'm pretty sure there is something up with the server, because when I logged the clients into a local account, they had no problems with browsing or other Internet services.

 

Shouldn't it be able to handle that many connections and more? I recently rebuilt the OD Master in order to change the old .local domain name to a fully qualified domain name. I thought that should have resolved the issue. What might I be missing?

Mac mini, OS X Mavericks (10.9.4)

Posted on Sep 3, 2014 2:28 PM

Close

Q: OD performance grinds to a halt with 44 student logins - What do?

  • All replies
  • Helpful answers

  • by Strontium90,Helpful

    Strontium90 Strontium90 Sep 3, 2014 4:01 PM in response to jeffzimmm
    Level 5 (4,077 points)
    Servers Enterprise
    Sep 3, 2014 4:01 PM in response to jeffzimmm

    Three thoughts.

     

    1:  Sketch out your network.  How do all the devices connect together.  For example, you list 2 Netgear switches.  Ok, if you have 30 of those client workstations coming into one switch and your server is on the other, then you are forcing 30,000Base over a 1,000Base switch to switch uplink (as those Netgears are likely not stackable).  I am not even going to talk about the wireless and the issues that can cause.  If those NetGears support link aggregation, you should get the mini up to 2000Base.  A Thunderbolt to 1000Base of USB3 to 1000Base adaptor will allow you to pull this off.  Move you server "closer" to the clients or increase your backhaul.

     

    2:  Where are your network home folders located?  If they are on the 5400 rpm internal drives of the Mac mini Server... well you have my sympathies.  You probably should look to get faster storage to help deliver the content faster.

     

    3:  Did you disable Spotlight, make sure no-one has fonts, redirected your cache and temp files?  All of these tricks can help ease the load when network homes are active.

     

    Reid

    Apple Consultants Network

    Apple Professional Service

    Author "Mavericks Server – Foundation Services" & "Mavericks Server – Control and Collaboration" :: Exclusively available in Apple's iBooks Store

  • by Linc Davis,

    Linc Davis Linc Davis Sep 3, 2014 4:36 PM in response to jeffzimmm
    Level 10 (207,995 points)
    Applications
    Sep 3, 2014 4:36 PM in response to jeffzimmm

    Many Open Directory problems can be resolved by taking the following steps. Test after each one, and back up all data before making any changes.

    1. The OD master must have a static IP address on the local network, not a dynamic address.

    2. You must have a working DNS service, and the server's hostname must match its fully-qualified domain name. To confirm, select the server by name in the sidebar of the Server application window, then select the Overview tab. Click the Edit button on the Host Name line. On the Accessing your Server sheet, Domain Name should be selected. Change the Host Name, if necessary. The server must have at least a three-level name (e.g. "server.yourdomain.com"), and the name must not be in the ".local" top-level domain, which is reserved for Bonjour.

    3. The primary DNS server used by the server must be 127.0.0.1 (that is, itself) unless you're using another server for internal DNS. The only DNS server set on the clients should be the internal one, which they should get from DHCP if applicable.

    4. Follow these instructions to rebuild the Kerberos configuration on the master.

    5. If you use authenticated binding, check the validity of the master's certificate. The common name must match the hostname and domain name. Deselecting and then reselecting the certificate in Server.app has been reported to have an effect in some cases. Otherwise delete all certificates and create new ones.

    6. Unbind and then rebind the clients in the Users & Groups preference pane. Use the fully-qualified domain name of the master.

    7. Reboot the master and the clients.

    8. Don't log in to the server with a network user's account.

    9. Disable any internal firewalls in use, including third-party "security" software.

    10. If you've created any replica servers, delete them.

    11. As a last resort, export all OD users. In the Open Directory pane of Server, delete the OD server. Then recreate it and import the users. Ensure that the UID's are in the 1001+ range.

  • by jeffzimmm,

    jeffzimmm jeffzimmm Sep 3, 2014 5:14 PM in response to Strontium90
    Level 1 (8 points)
    Servers Enterprise
    Sep 3, 2014 5:14 PM in response to Strontium90

    Re: Network topology - Right now, the switches, wireless AP, and Mac mini are all on the router. The switches are not daisy-chained. I'm not sure how I could move the clients closer to the server than that, save putting it directly on one of the two 24-port switches, but then that would makes things worse for clients on the other switch.

     

    Re: Home folders - Yup, they're on the slow Mac mini hard drive. Our old PowerMac, OS X 10.6 server at least had 7200rpm drives. Maybe I underestimated the harm that would do.

     

    Will definitely try those tricks.

  • by Strontium90,Solvedanswer

    Strontium90 Strontium90 Sep 3, 2014 6:19 PM in response to jeffzimmm
    Level 5 (4,077 points)
    Servers Enterprise
    Sep 3, 2014 6:19 PM in response to jeffzimmm

    So your network is a tough one.  You need a good backbone and as fast of performance on the mini as you can get.  I am a fan of 10GigE for network home folder servers but not everyone has the budget for that.  I also will split services, isolating the network home servers on independent hardware.  Where I am given the opportunity I will even get class schedules and try to split students across multiple servers, ensuring that the load is split to multiple machines. 

     

    Since the year is already going, 10GigE is out of the question.  Look into link aggregation.  A $40 adaptor is affordable and you can double the mini's network connection.  The more network you can provide the better.

     

    The setup of your devices is a bit more challenging.  If you have 20 devices connected to a switch that has a single 1000Base uplink to another switch and your server is on the second switch, then when all devices request data at the same time, you basically relegate each of the 20 devices to 50Base across the switch uplink.  Now, add in the requests from the devices that are on the same switch as the server and you have a serious over-subscription problem. 

     

    Ouch on the home folder location.  Ya, those internal drives are really not up to the task.  Even if you have one for the OS and the other for the network home folder, you are likely hitting disk throughput on them.

     

    Hope this helps you out.  I will confirm that network home folders are a viable solution to environment that can not employ one-to-one deployments.  But your infrastructure must be rock solid to support the demands of 30+ devices all logging in at the same time.

     

    Reid

    Apple Consultants Network

    Apple Professional Service

    Author "Mavericks Server – Foundation Services" & "Mavericks Server – Control and Collaboration" :: Exclusively available in Apple's iBooks Store

  • by jeffzimmm,

    jeffzimmm jeffzimmm Sep 3, 2014 9:12 PM in response to Strontium90
    Level 1 (8 points)
    Servers Enterprise
    Sep 3, 2014 9:12 PM in response to Strontium90

    Can you help me find some of these link aggregation adapters? My Google-fu is weak in this case. Can they add LAG to a switch that did not already have it?

     

    Also, check my understanding here. Supposing I had two 24-port gigabit switches with LAG. I would have to change the configuration to get the most benefit. Right now it's:

     

    Router_____________________________________

    /               \                         \                         \

    Server     Wireless-AP     24portswitch     24portswitch

                        \                         \                         \

                   Clients               Clients               Clients

     

    With link aggregation, it should look more like this:

     

    24portswitch____________________________________________

    //               \               \\                         \                         \

    Server     Router     24portswitch     Wireless-AP     Clients

                                       \                         \

                                  Clients               Clients

  • by Strontium90,

    Strontium90 Strontium90 Sep 4, 2014 3:54 AM in response to jeffzimmm
    Level 5 (4,077 points)
    Servers Enterprise
    Sep 4, 2014 3:54 AM in response to jeffzimmm

    Sure.  If you have nothing connected to the Thunderbolt port, you can get Apple's:

     

    http://store.apple.com/us/product/MD463ZM/A/thunderbolt-to-gigabit-ethernet-adap ter

     

    However, since the mini has only 1 TB port, I tend to try and keep that open.  I've had success with:

     

    http://www.siig.com/usb-3-0-to-gigabit-ethernet-adapter.html

     

    Configuring the bond on the Mac side is as easy as going into Network Preferences and creating a new virtual interface. 

     

    On your switch side, you need to check your switch models and see what features it supports.  There is this link from NetGear:

     

    http://kb.netgear.com/app/answers/detail/a_id/21632/~/what-are-link-aggregation- groups-(lags)-and-how-do-they-work-with-my-managed

     

    The Router is providing internet and your primary concern is the LAN.  If the Router is just a WAN and LAN port type of device, then you have one cable to feed the LAN.  Define one switch as the "main" and connect it.  Then create a LAG connection between the two switches assuming you have enough ports.  You should be able to increase in pairs.  So a 2000, 4000Base, etc.  Your drawing is looking about the best you can do.

     

    Now, once you have the switches pair bonded, you have more throughput between the devices.  Then select an additional port pair to allow the mini to also get a faster connection as you have shown.

     

    More advanced switches has special stacking ports on the back that make this unnecessary.  Using up front ports simply takes away from the capacity of the switch and only grows arithmetically.  Stacking ports are often in the order of 40Gb, allowing nearly the entire fabric of one 48 port switch to trunk to the other.  Then you can get into aggregating server across switch nodes to produce even better redundancy.

     

    Reid

    Apple Consultants Network

    Apple Professional Service

    Author "Mavericks Server – Foundation Services" & "Mavericks Server – Control and Collaboration" :: Exclusively available in Apple's iBooks Store

  • by jeffzimmm,

    jeffzimmm jeffzimmm Sep 4, 2014 11:01 AM in response to Strontium90
    Level 1 (8 points)
    Servers Enterprise
    Sep 4, 2014 11:01 AM in response to Strontium90

    This has been one of the most educational support requests I've ever had. Thank you.

     

    Sadly, our switches are JGS524s, which I see from the specs do not have LACP. But the JGS524E seems like an inexpensive way to get that feature. Couple of those, plus an external 1TB SSD should at least double our bandwidth, all for less than $1k. The secondary 5200rpm internal drive in the Mac Mini can be used for backup

     

    But, if there's money in the budget (probably in the $5000 range), I might be able to snag a couple 10GbE switches and a 10GbE Thunderbolt adapter for the Mac Mini, plus the external SSD. That would make a difference.

  • by Strontium90,

    Strontium90 Strontium90 Sep 4, 2014 1:28 PM in response to jeffzimmm
    Level 5 (4,077 points)
    Servers Enterprise
    Sep 4, 2014 1:28 PM in response to jeffzimmm

    Glad to help.  As you might guess, I do this frequently and I feel your pain.  Education is often thrust to the bleeding edge of technology but infrastructure is often hobbled together with components from the stone age.  I am going to guess that "taking care of the computers" is your "part time" responsibility.  Too often I see teachers burdened with the responsibility of IT services but it just ends up diluting the efforts of both teaching and IT services.  Hang in there.  You are not alone.

     

    By the way, if you are eligible for E-Rate purchasing, you might want to reach out to Extreme Networks or Cisco.  You likely can get enterprise class switches at very favorable pricing.  These devices will support stacking, LACP, and will have a backplane that exceeds the aggregate port capacity.

     

    Good luck with your investigations.