safdwewwe
Level 1 (9 points)
Desktops

Q: syncing RAIDs at 2 different locations

I recently started working in 2 different locations and need to figure out a way to keep my 15TB+ of continuously changing and growing data (photoshop files in RAID 0 formation) handy at both locations at all times.

 

My best idea so far is to get a transportable 15TB work drive that I bring with me back and forth and then keep a stationary TIME MACHINE RAID at each location that continuously back up this transportable work drive...

 

If anyone has a better idea, please advise.

Posted on Sep 25, 2016 12:26 PM

by John Lockwood,Solvedanswer
John Lockwood John Lockwood
Level 6 (9,411 points)
Servers Enterprise
A:

Firstly lets get the RAID issues out of the way. As Grant said a RAID0 setup does not give you any protection at all, in fact strictly speaking it is twice as likely to fail as a single drive. RAID0 merely gives you a faster system. There are various other RAID systems which can give you protection, for example RAID10 is a combination of RAID0 and RAID1 effectively you get both a stripe set for speed, and a mirror for reliability it would need four drives to accomplish this. There is also RAID5 which uses typically four hard disks one of which is a backup for the other three and therefore from a space point of view more efficient than RAID1, it is however slower.

 

Now next synchronising data between two sites. What a lot of people forget regarding this is that data can be modified at both ends, and potentially at the same time. Some synchronisation products can only really cope with syncing from A to B and not both directions at the same time.

 

One approach I suspect will not be suitable to you is Microsoft has a feature built-in to their file server software called DFS - Distributed File System. This is specifically designed to allow the same data to be accessed at multiple sites and for each site to get full local speed with both sites kept in sync automatically. This would require Windows file servers at both sites. Macs can access DFS but cannot act as DFS servers. It should also be possible to use a Linux server running SAMBA4 as a DFS server.

 

There are also various NAS servers with built-in synching capabilities, they are harder to impossible to fully integrate with Open Directory but for a simple generic file server are good enough. NAS boxes also offer various levels of RAID as well so this would kill two birds with one stone.

 

A larger enterprise might use a separate data centre to which both offices are directly linked and therefore both offices see the same file server contents. I get the impression this would not be describing your situation.

 

Finally there are various software programs you could run on a Mac server to sync to another Mac server but bear in mind my comment about bi-directional syncing. There is Rsync a version of which is built-in to OS X, this could be scripted to run several times a day automatically, there is also ChronoSync. Since rsync is a built-in command line (Unix) tool run Terminal and type man rsync for information and/or Google for advice on scripting it.

 

A final reminder, a typical home broadband link has a fast download connection but a much, much slower upload link. This could make it painfully slow to send files to the other site.

Posted on Sep 26, 2016 7:21 AM

Close
safdwewwe

Q: syncing RAIDs at 2 different locations

  • All replies
  • Helpful answers

  • by Grant Bennet-Alder,

    Grant Bennet-Alder Grant Bennet-Alder Sep 25, 2016 1:08 PM in response to safdwewwe
    Level 9 (61,390 points)
    Desktops
    Sep 25, 2016 1:08 PM in response to safdwewwe

    Keeping your Precious data on a Striped RAID is dangerous. Striped RAID sometimes adds speed, but in case of drive failure, is likely to result in a total loss of all data with no hope of partial recovery.

     

    Putting Time Machine data on a Mirrored RAID is a waste of drives.

     

    Time Machine can automatically Alternate drives to keep two (or more) independent Backup sets. Just add the second drive and it automatically alternates.

     

    As always, you can use AppleRAID or an external RAID enclosure for building a concatenated RAID set that looks like one drive to Mac OS X.

  • by safdwewwe,

    safdwewwe safdwewwe Sep 25, 2016 6:33 PM in response to Grant Bennet-Alder
    Level 1 (9 points)
    Desktops
    Sep 25, 2016 6:33 PM in response to Grant Bennet-Alder

    hmmm.. don't think you read my question...

  • by Grant Bennet-Alder,

    Grant Bennet-Alder Grant Bennet-Alder Sep 25, 2016 7:05 PM in response to safdwewwe
    Level 9 (61,390 points)
    Desktops
    Sep 25, 2016 7:05 PM in response to safdwewwe

    I read your question and understood it as far as you explained it.

     

    You have a very difficult problem needing to have so much data available, and there are no "magic bullet" solutions that I know of beyond what you are already planning to do. Moving 15TB of data across the Internet in under an hour is out of the question. The only other solutions require the drives to be present at their location of use.

     

    My response was limited to the portions of your proposed solution that gave me the creeps.

  • by John Lockwood,Solvedanswer

    John Lockwood John Lockwood Sep 26, 2016 7:21 AM in response to safdwewwe
    Level 6 (9,411 points)
    Servers Enterprise
    Sep 26, 2016 7:21 AM in response to safdwewwe

    Firstly lets get the RAID issues out of the way. As Grant said a RAID0 setup does not give you any protection at all, in fact strictly speaking it is twice as likely to fail as a single drive. RAID0 merely gives you a faster system. There are various other RAID systems which can give you protection, for example RAID10 is a combination of RAID0 and RAID1 effectively you get both a stripe set for speed, and a mirror for reliability it would need four drives to accomplish this. There is also RAID5 which uses typically four hard disks one of which is a backup for the other three and therefore from a space point of view more efficient than RAID1, it is however slower.

     

    Now next synchronising data between two sites. What a lot of people forget regarding this is that data can be modified at both ends, and potentially at the same time. Some synchronisation products can only really cope with syncing from A to B and not both directions at the same time.

     

    One approach I suspect will not be suitable to you is Microsoft has a feature built-in to their file server software called DFS - Distributed File System. This is specifically designed to allow the same data to be accessed at multiple sites and for each site to get full local speed with both sites kept in sync automatically. This would require Windows file servers at both sites. Macs can access DFS but cannot act as DFS servers. It should also be possible to use a Linux server running SAMBA4 as a DFS server.

     

    There are also various NAS servers with built-in synching capabilities, they are harder to impossible to fully integrate with Open Directory but for a simple generic file server are good enough. NAS boxes also offer various levels of RAID as well so this would kill two birds with one stone.

     

    A larger enterprise might use a separate data centre to which both offices are directly linked and therefore both offices see the same file server contents. I get the impression this would not be describing your situation.

     

    Finally there are various software programs you could run on a Mac server to sync to another Mac server but bear in mind my comment about bi-directional syncing. There is Rsync a version of which is built-in to OS X, this could be scripted to run several times a day automatically, there is also ChronoSync. Since rsync is a built-in command line (Unix) tool run Terminal and type man rsync for information and/or Google for advice on scripting it.

     

    A final reminder, a typical home broadband link has a fast download connection but a much, much slower upload link. This could make it painfully slow to send files to the other site.

  • by safdwewwe,

    safdwewwe safdwewwe Sep 26, 2016 7:21 AM in response to John Lockwood
    Level 1 (9 points)
    Desktops
    Sep 26, 2016 7:21 AM in response to John Lockwood

    Thank you so much for this!  In the past, I've used RAID 5, but found it too slow.  My archive consists of hundreds of thousands of images that I continuously pull from and alter so I need maximum speed.  I considered RAID 10, but relying solely on one system gave me the creeps, so what I did instead was create a separate unit also in RAID 0, as a time machine backup.  Then I keep two backups off-site that I alternately update monthly.  The one time I needed to use time machine though, time machine for whatever reason had erased all my backups, so I'm not entirely happy with this solution.  Luckily my off site copy was recent enough so I didn't lose any important data.  T

     

    Because I'm the only user, I'm not too worried about data changing at both sides simultaneously, unless of course I forget to update myself and confuse things that way.  I will research your other options and see if any of them work. The separate locations are 2000 miles apart and I wouldn't be able to keep things online at both sites simultaneously. 

  • by John Lockwood,Helpful

    John Lockwood John Lockwood Sep 27, 2016 6:52 PM in response to safdwewwe
    Level 6 (9,411 points)
    Servers Enterprise
    Sep 27, 2016 6:52 PM in response to safdwewwe

    RAID should never be looked at as a backup solution, so if you used RAID10 it would give you a combination of speed and protection against a hard disk failure. However you should continue to use something like Time Machine for backups.

     

    The benefit of the RAID10 is that if a drive failed you would have zero downtime whereas currently you would have to replace the dead drive and do a full 100% restore from your Time Machine backup to rebuild the RAID0 which would take ages.