This discussion is locked
4fx

Q: Could Xsan performance degradation be caused by an external network?

We have a 2 client Xsan setup that we have been using for almost a year now. Within the last month or so, we have started getting dropped frames when ingesting footage. I ran Xsan Tuner to see if the SAN was the hiccup and sure enough it is comming up with dropped frames on the FCP write test. Reading seems to be fine.

Whats interesting is that when performing the write test, the current bitrate fluxuates by small amounts continuously until it drops below the miniumum (this goes for both MiniDV and Uncompressed 8bit, both of which we use).

I remember reading a while ago about external network connections causing problems, and I tried running the test with only the metadata network active. In this case, the current bitrate remains constant at the target rate and there are no dropped frame warnings.

Any ideas what I can do to solve the problem? A temporary workaround can be to turn off the network connection whenever I am ingesting footage, but this is a hassle. Im not sure if the IT department has modified anything in the last month or so. Im going to try to get ahold of them to see.

PowerMac G5, Dual 2.5GHz   Mac OS X (10.4.8)  

PowerMac G5, Dual 2.5GHz, Mac OS X (10.4.6)

Posted on Nov 20, 2010 7:54 AM

Close

Q: Could Xsan performance degradation be caused by an external network?

  • All replies
  • Helpful answers

  • by Mark Raudonis,

    Mark Raudonis Mark Raudonis Nov 20, 2010 7:55 AM in response to 4fx
    Level 1 (70 points)
    Nov 20, 2010 7:55 AM in response to 4fx
    You say it's been working for the past year. What's different now? New equipment? Software change or upgrade? Storage filled past "full"?

    If it WAS working before, then there's something new in your set up that's conflicting. Dropped frames are typically a "connectivity" issue.

    It could be that your meta data network is getting "confused" by your external network connection. Try unplugging your "external" network, then restarting.
    Once you've established connection to the SAN, THEN plug in your external network. See if that makes a difference.

    If that doesn't do it, this is not something that you can easily troubleshoot on a forum. Call in an expert to check your settings and connectivity.

    Good luck.

    Mark
  • by Arnór Kristjánsson,

    Arnór Kristjánsson Arnór Kristjánsson Nov 20, 2010 7:55 AM in response to 4fx
    Level 1 (40 points)
    Nov 20, 2010 7:55 AM in response to 4fx
    Would identical UIDs (i.e. not using a directory server or not making sure UIDs are unique) on clients be able to cause slowdowns?
  • by RobertKite,

    RobertKite RobertKite Nov 20, 2010 7:55 AM in response to Arnór Kristjánsson
    Level 1 (120 points)
    Nov 20, 2010 7:55 AM in response to Arnór Kristjánsson
    Do you have local accounts or network accounts? Where are the home folders for the users?
  • by Ivailo Djilianov,

    Ivailo Djilianov Ivailo Djilianov Nov 20, 2010 7:55 AM in response to 4fx
    Level 1 (25 points)
    Nov 20, 2010 7:55 AM in response to 4fx
    do logs show anything strange?
  • by 4fx,

    4fx 4fx Nov 20, 2010 7:56 AM in response to 4fx
    Level 1 (80 points)
    Nov 20, 2010 7:56 AM in response to 4fx
    It doesnt look like the external network could be causing this. Ive tried disconnecting it from all of the SAN equipment and it doesnt seem to make any difference.

    I double checked the settings on the sanbox 5200 and everything seems to be correct.

    To answer previous questions, the user folders are stored locally and we are not using OD since there are only 2 clients and it was therefore easier in my opinion to set up the UIDs manually. There is approximately 50% free storage space.

    Server CPU and memory usage appears to be minimal.

    We have 4 LUNs total, 1 for metadata, 2 for main video storage, and 1 as a render pool. Writing to the Render pool is relatively fast, but tests using Xsan Tuner still show an occasional dropped frame (doesnt matter what codec). This happens more frequently on the video pool.

    Also, copying files to the video pool from the finder has gotten worse in the last week. It can now take 2 or 3 minutes before any data is written to the SAN (small or large files have the same thing happen). And write performance is even worse now. Im surprised that Xsan Tuner can even do a test at all.

    Reading appears to be as fast as ever with no hiccups (that Im aware of).

    I checked the log, but am unsure what to make of it. However, I am getting the following every 1 minute and 1 sec:

    [0118 15:29:07.350340] 0xa000ed88 (Debug) Node [41] [10.1.0.1:50173] connected.
    [0118 15:29:07] 0x19c4200 (Info) Node [41] [10.1.0.1:50173] Administrator Logout.
    [0118 15:29:07.369344] 0xa000ed88 (Debug) Node [42] [10.1.0.1:50175] connected.
    [0118 15:29:07] 0x1908000 (Info) Node [42] [10.1.0.1:50175] Administrator Logout.
  • by 4fx,

    4fx 4fx Nov 20, 2010 7:56 AM in response to 4fx
    Level 1 (80 points)
    Nov 20, 2010 7:56 AM in response to 4fx
    At this point we are ready to have an expert come out and assist us get up and running again.

    If anyone has any suggestions of where I could find someone in southern CA, I would appreciate it greatly. Ive tried to search, but havent been successful.

    PowerMac G5, Dual 2.5GHz   Mac OS X (10.4.6)  
  • by Mark Raudonis,

    Mark Raudonis Mark Raudonis Nov 20, 2010 7:56 AM in response to 4fx
    Level 1 (70 points)
    Nov 20, 2010 7:56 AM in response to 4fx
    I would recommend Shane Sokolosky.

    Shane Sokolosky
    Circuit Access

    Email - heysomthin@aol.com
    Phone - (714) 599-1611
  • by 4fx,

    4fx 4fx Nov 20, 2010 7:56 AM in response to 4fx
    Level 1 (80 points)
    Nov 20, 2010 7:56 AM in response to 4fx
    Last night I discovered that one of our storage pools (used for rendering and temporary compression files) was not affected by any sort of slowdown like our main storage pool. Since that particular storage pool was only associated with 1 LUN which exists on the lower controller of the RAID, I got to thinking that maybe the problem had to do with the upper raid controller.

    The 3 items that I could think of that could cause the problem were bad raid controller, bad cable, or bad port on the fibre switch. Since we had several extra copper cables lying around, I thought I would start there.

    Sure enough, after replacing the cable write speeds are now back to normal!

    Im still a little confused about how I was able to write to the storage pool at all, albeit at an incredibly slow rate. I assumed if there was a cable problem, it would be an all or nothing thing.

    PowerMac G5, Dual 2.5GHz   Mac OS X (10.4.8)