using two Mac Mini servers as a mirrored pair for redundancy

the modalman wrote:
I run a small business (10+ users) that has a mixed environment of macs (all running SL) and PCs (running XP and WIN 7) about 50:50. We currently use Windows SBS 2003 and outlook- we run two parallel systems in case one goes down (our data is our livelihood) and have mirrored RAIDS on both that currently only mirror every few hours. I am sick of running giant noisy energy hungry servers and having to pump tons of regular maintenance to keep the SBS running and it has become more unstable lately so would need to buy new SBS server. I am thinking of dumping it all and going to gmail and SL mac Mini Servers. We back up off site every night and locally need a RAID server with approx 300GB of data stored for daily active use. In my ideal world I would like to have two identical SL Mac Minis running in parallel and ideally up to date with each other in real time (although a reasonable time period of several hours is OK). In the event of failure it would ideally fail over automatically or simply require an admin person to log in and just select a swap over with a simple step (or set of steps) I do not want to rely on always having my remote IT person doing it. I have searched high and lo and have not found a good posting from anyone who has implemented this kind of set up. It seems such a straightforward idea - to have two parallel system running with a simple fail over - and with Mac Minis being so inexpensive.... Can anyone advise? thanks

The typical way of doing this (on Mac servers) is to have the servers configured in IP failover mode. Typically also you have the data stored externally to the servers on a shared SAN (Storage Area Network) which is protected by RAID so the (protected) data is common to both and does not need to be synchronised. The servers would be configured with the same settings and due to the IP Failover setting if the primary fails the secondary takes over automatically.

However, typically, the two servers would be connected together via a second separate LAN to do the IP Failover handshaking, and also typically would use Fibre-Channel to connect to the SAN, and possibly also require and use XSan.

As far as I am aware Apple have not implemented support for the simpler (and far cheaper) option of creating a 'poor man's SAN' via iSCSI over Ethernet. It is possible to use iSCSI on Macs, there are several iSCSI initiators available for Mac so it might be possible to do it ones self. As examples the ReadyNAS Pro supports acting as an iSCSI server, as does the Drobo Elite and Drobo Pro. These would also solve the limitation of the Mac mini to currently 500GB drives.

While a Mac Pro or XServe both have dual Ethernet ports and PCI-E slots to allow adding a Fibre-Channel connection, the Mac mini does not. You could use if needed a USB to Ethernet adaptor, if the IP Failover does need a separate network then the traffic should be low enough for the slower USB connection to not matter, you could also use TCP/IP over FireWire to link the two together.

There is more info in the Mac OS X Server manuals but also have a look at this Apple article http://docs.info.apple.com/article.html?path=ServerAdmin/10.5/en/c3fs29.html it should still apply to 10.6.

Live failover is complex. If you don't need it here's what I would do:

Setup timemachine (or some other backup solution) to do a backup of the system every hour.

Keep your data files (non-system files, basically all your file shares) on a separate external RAID system, and back it up separately to a separate local drive and also to an off-site drive.

If your primary system fails, restore your second standby system off of the hourly backup. This is easy to do, you could write down the steps so that several of your staff are capable. Once the restore is finished (if your user data is elsewhere, it could be done in less than an hour) have them physically replace the broken machine with the new one, boot it up, and you're off.

If your external RAID fails with the data on it, just drop in your local backup and order a new RAID on Monday morning. Also easy to do.

--
John

PS. Make sure you figure out what things need non file-level backup. For instance my postgreSQL database files can't be relied on for a backup unless the database system is stopped, so I backup the database separately nightly using the approved postgres command to dump a database. Also my virtual hard disk for VirtualBox which hosts a virtual WindowsXP system is one big gigantic file and I don't want to back it up hourly, so I exclude it from timemachine and have a script to back it up nightly.

PPS. I wonder if there's a way to restore a computer remotely, without going into the office? I think if I could connect to my VPN I could do it with a SLServer remote install. But then my VPN would have to be managed separately from the dead server (perhaps a router-based VPN), and also the system backup would have to be on a separate machine still attached to the network, e.g. on the same RAID server that has the fileshare data.

No disrespect, but your major problem is that you've hybrided your present setup and are trying to duplicate it when your original design was faulty.

A 10-20 user environment does not need to run failover systems, if the servers were properly configured in the first place. First, in data centers, mirrored drives are never used for databases. Mirrored drives are used for operating systems because they help accelerate accessed to the operating system. Databases are placed on Raid-5, or equivalent raided, drive systems with at least 3 drives (depending on size we might use 5 or more). This allows full redundancy if a drive fails as you can hot swap out the bad drive and the controller will automatically rebuild it. The only thing that will happen is that the Raid array output might slow down as it has to create the missing data from the other working drives through the checksums stored across the raid. Once the drive is rebuilt you will get your original speed back (this problem becomes very minor with a 4 or more drive raid system).

The DroboPro under iSCSI is a great drive to use for database operations as it is a modified Raid-5 system with all drives hot swappable. I've been using one for 6-months now, have had one drive failure, and no fallout from using it for all my data needs. the iSCSI speed is blazingly fast. The company it's being used at is a multi-million dollar multi-national firm where data protection is extremely important.

For servers I've been using a pair of mac-minis, in a master/replica setup. The services have been structured to provide fastest service to the clients, with the Master serving OD, AD, dns and dhcp. The replica is connected to the DroboPro and provides mail, web, and all data share and storage control, including running the backup systems. All mac clients are bound to the replica. Both mac minis contain the dual 500G drives which have been set up as mirrored drives.

IP failover is expensive and, if used, can be found in nocs that support hundreds to thousands of users (such as Microsoft, AT&T, Chase, etc.). However, most large DCs used load-balancing systems (F5), mirrored drives on the servers (many are 1U such as HP DL360's), and SANS for databases, but no fail-overs, except in for clustered database servers.

I can understand your problems with SBS. It is designed for light to medium usage single server networks. It does not support multiple server domains or multi-domain systems as SBS has to be the only domain server. If you feel forced to stay with Windows then your best option would be to look into Essentials Business Server which requires 3 x64 servers and sets them up in a semi-clustered environment (exchange and ISA is load balanced across two of them). For your purposes, however, I'd suggest looking into a structure similar to the one I installed for my client.

pcolvin15 wrote:
For servers I've been using a pair of mac-minis, in a master/replica setup. The services have been structured to provide fastest service to the clients, with the Master serving OD, AD, dns and dhcp. The replica is connected to the DroboPro and provides mail, web, and all data share and storage control, including running the backup systems. All mac clients are bound to the replica. Both mac minis contain the dual 500G drives which have been set up as mirrored drives.

Can you point us to more resources on "a master/replica setup". A quick google search suggests this is only for LDAP/OpenDirectory. I was hoping this meant that the full server OS and config files would be live replicas of the master, but I guess that's too much to hope for?

Also the to OP: don't trust RAID to protect your data, they only protect against one kind of common failure -- a hard drive failure. Keep an on-site backup and an off-site backup of anything important.

You're right in that master/replica would be for the OD. Information can be found in the SLS Open Directory Admin Manual. However, in regards to DNS, you can have a master/secondary (the OD replica carries the secondary zone [SLS Network Services Manual p50]). For DHCP only one box runs it. For proper setup I'd refer to Mr. Hoffman's detailed instruction sets to help avoid the pitfalls. Binding the mac clients to the replica I found buried in an Apple KB article.

Needing a reliable backup for any system, including a single laptop/desktop, is a given. Backups should be scheduled and the backup media held on-site/off-site according to a rotational schedule. Daily backups usually are incrementals, weekly and monthly fulls, and, at minimum, the monthlys should be taken off-site and stored for months before being rotated back for re-use (if you are using r/w media. I use BRU and Tandberg removable hard drives as my backup system because BRU has agents for both OSX and Windows. If you are running any sort of a database server then you may want to consider adding a db agent into the licensing mix, otherwise you will have to shut off the db engine in order to get a backup of the database.

I hate to disagree, but the people at Data Robotics just don't know what they're doing. Half the time they come out with a firmware update for their products, it takes down the Drobos, requiring a call to technical support. The last update a few days ago that we applied to an OS X Server running 10.6.6 took down TWO different Drobo systems, and the systems reported drives had failed when they had not. Use Drobo products as redundant storage only, but never as primary storage.

More importantly, stay away from RAID5 for larger drive systems. These days, if you suffer a drive failure on a RAID5 built on larger drives, your chances are greater than 50% that another drive in the array will fail DURING the rebuild, and you could lose all your data. Stick with RAID6 (dual drive redundancy) or use the dual-drive redundancy option on the Drobo to protect your data.