Q: Xsan 4 Spotlight MDChannel errors

After upgrading to Yosemite (v.10.10.4) and Xsan 4 (previously running Lion w/ Xsan v.2.3), we are seeing consistent errors related to searching that SAN volume. Specifically, when using either the Spotlight menu, a Finder window, and/or `mdfind`, we experience errors on the client-side that stops the search from continuing or returning a full list of results. At the time that the problems are encountered, the Console on the client logs the following errors:

7/31/15 3:28:14.762 PM mds[58]: XSANFS_FSCTL_SpotlightRPC fsctl failed (errno = 7)

7/31/15 3:28:14.763 PM mds[58]: ERROR: _MDSChannelXsanRequest: _XsanMDSChannelRequest failed: 7

7/31/15 3:28:14.763 PM mds[58]: (Message.Error:142) MDSChannel RPC failure (fetchQueryResultsForContext:)

7/31/15 3:28:14.763 PM mds[58]: (Store.Error:273) <MDSDistantStore 0x7fb1a8f74ae0 shutdown:NO got shutdown notification:NO>{channel:0x7fb1b61e2ac0 localPath:'/Volumes/CreativeSAN'} MDSChannel failed -- initiating recovery

When performing Spotlight queries directly on the primary MDC, we are seeing that all lookups can be performed without error (i.e., all results are provided, and no MDChannel errors are logged in Console.) To try to correct the issues, we've confirmed that the Spotlight index was rebuilt completely on the master by disabling Spotlight (`mdutil -i off /Volumes/CreativeSAN`), removing the Spotlight index (`rm -rf /Volumes/CreativeSAN/.Spotlight-V100`), and then re-enabling the service (`mdutil -i on /Volumes/CreativeSAN`). During the index rebuild, the SAN volume is not mounted on any clients (including the secondary MDC) as recommended in this article: Xsan 2: Clients cannot search volumes mounted before indexing was enabled - Apple Support ... After that rebuild, the client's Spotlight searches result in the same errors.

To try to confirm that permissions problems aren't affecting the lookup, we've attempted to use a combination of ACLs (which were previously being used to manage permissions), as well as a custom umask (to force POSIX permissions to more relaxed values.) In either situation, we get the same results. The ACE that we've applied to that volume is a "Full Control" entry for the "Everybody" group (which has worked just fine in the past, and continues to allow the users on the volume to collaborate.) Additionally, we've set POSIX permissions on all files/folders to Read&Write (`chmod -R 777 /Volumes/CreativeSAN`), which didn't seem to help. Even though the problems don't appear to be permissions related (the index works normally on the PMDC), we've attempted to apply an Access Control Entry for the Spotlight process (to allow it access to all files/folders in the volume) without a change in behavior (`chmod -R +ai "_spotlight allow list,search,file_inherit,directory_inherit`). We've attempted to switch to using POSIX permissions in combination with a custom umask (as opposed to an ACL) without a change in behavior (using recommendations from this article: Setting a custom umask in OS X - Apple Support).

It's worth noting that we were able to perform searches on the same data using Lion clients/MDCs. Although the Spotlight index would get corrupted in the past, a simple rebuild (unmount volume from clients, reindex using PMDC) always worked.

One additional piece of information is that when adding the SAN volume as an AFP share, and performing Spotlight searches on the mounted AFP volume, we're seeing similar errors being reported. The only difference is that the function call for the MDSChannel request references AFP instead of Xsan (_MDSChannelAFPRequest vs _MDSChannelXsanRequest)

Any suggestions are welcome and appreciated.