Taming Spotlight

Mac OS 10.4.11 on an iMac g3 600.

( It's the Tiger forum. Does the machine make a difference? )

Robert

No.

I got the impression that there was one index for the whole machine. The index covers multiple users, so just because one users say no for an extension, another user could say yes. I've always done the ordering of the search results on an admin id. I'll have to try it from a standard id.

Does that get the files out of search? That would be great.

I dropped several folders on the privacy tab.

Robert

I'm almost sure there is one index. The filtering is down on the search. Not too secure of a design, but I guess you need it to avoid multiple indexes of the same thind.

Here is what I see:

mac $ sudo bash
Password:
in /Users/mac/.bashrc       but claims to be in bash
in /Users/mac/config/profile.bash     claims to be bash
root # ls -lad /.Spotlight-V100/
drw-------   9 root  admin  306B May 11 12:03 /.Spotlight-V100//
root # cd .Spotlight-V100/
root # pwd
/.Spotlight-V100
root # ls -la
total 575080
drw-------    9 root  admin       306B May 11 12:03 ./
drwxrwxr-t   49 root  admin         1K Dec 31  1969 ../
-rw-------    1 root  admin         0B May 11 13:43 .journalHistoryLog
-rw-------    1 root  admin        44M May 11 13:44 .store.db
-rw-------    1 root  admin       192M May 11 13:04 ContentIndex.db
-rw-------    1 root  admin        55B Apr 27  2012 _IndexPolicy.plist
-rw-------    1 root  admin       456B May 11 12:03 _exclusions.plist
-rw-------    1 root  admin       378B Apr 27  2012 _rules.plist
-rw-------    1 root  admin        44M May 11 13:44 store.db
root #

Some info on how exclusions work:

http://mactechnotes.blogspot.com/2005/10/spotlight-control.html

I'll have to play around with this. In the next week or so.

Robert

fyi:

I'm playing around with spotlight. I am using spotless to stop spotlight and delete the data base. I'm changing _rules.plist.

To see if my changes work, I create a new file or send a new email. Spotlight indexes these changes immediaely. It doesn't have to index the entire volume. The changed files and email will show up soon.

Created a new document called robertesting

Dont know why the new email didn't show up in tis list. Mail find did show the new email.

Alwasys wondered how to test what folders are being indexed.

Answer: place a file with a unique name on all directores. Search for it!

Robert

the other logged in users will continue indexing the drive you added.

You should not think of it as other users. X does this in the background on spotplight processes. See activity monitor. Set to see all processes. All the processes start with md... The individual users communicate with the spotlight processes to tell it what to do. I'd assume all work is down on the md processes. Even the query. Unix runs processes. The user starts & stops processes. When the user logs in. The user gets a process.

That's my take on things.

Do you need to be admin to do this? Should be.

--------------------------------------------------------------

run fslogger to see what data spotlight gets on changed files.

http://osxbook.com/software/fslogger/

--------------------------------------------------------------

Wondering if that may be a clue to some people all of a sudden not being able to search? Or Spotlight going on forever & related problems?

verify that md... processes are running. check for crashes. It's hard to look at spotlight files. Need terminal. Unless you changed permission on files like it have done.

You would think the exclude would happend immediately. I bet you could find some security hole here. Would be ineresting if you could see a new file added to the excluded disk.

I noticed in Tiger, a standard user could see personal folders excluded by an admin user . Security hole in my view.

Robert

What file are we talking about? You mean the folder?

mac $ ls -lad  /.Spotlight-V100/
drwxrwxrwx   9 root  admin  306B May 15 11:27 /.Spotlight-V100//
mac $ ls -lah  /.Spotlight-V100/
total 60520
drwxrwxrwx    9 root  admin      306B May 15 11:27 ./
drwxrwxr-t   48 root  admin        1K Dec 31  1969 ../
-rw-------    1 root  admin        0B May 13 23:52 .journalHistoryLog
-rw-------    1 root  admin        8M May 15 13:18 .store.db
-rw-------    1 root  admin       12M May 14 13:16 ContentIndex.db
-rw-rw----    1 root  admin      238B May 13 23:37 _IndexPolicy.plist
-rw-rw----    1 root  admin      609B May 15 11:27 _exclusions.plist
-rw-rw----    1 root  admin      439B May 13 19:19 _rules.plist
-rw-------    1 root  admin        8M May 15 13:18 store.db

I noticed my folder time/date doesn't agree with the time/date on the store.db file.

I do know the folder time/date changes when time/date of a file in the folder changes. I do not know the details. I did add a file to the folder & the folder time/date was change. So to when I removed the file. Maybe the time/date of the folder changes at open of a file for write and not close.

I pasted together an Applescript to write a small file to each folder in a list of folders. This way we should be able to see what folders spotlight is indexing.

(* 

Place marker files in a list of folders so we can "see" what spotlight does.


    Author: rccharles
    
    For testing, run in the Script Editor.
      1) Click on the Event Log tab to see the output from the log statement
      2) Click on Run
      
      
    For running shell commands see:
    http://developer.apple.com/mac/library/technotes/tn2002/tn2065.html
    
    Copyright 2013 rccharles
    GNU General Public License 
    
    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation,  version 3
    
    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    For a copy of the GNU General Public License see <http://www.gnu.org/licenses/>.

    


 *)


on run
    
    global firstRunning
    
    -- Names of the folders in which to write a marker file.  Use Unix names path.
    -- find -x /   -type d -maxdepth 1 
    set folderList to ¬
        {¬
            "~/spotNoWrite", ¬
            "~/Desktop/Bethesda", ¬
            "~/sNoneForSpot", ¬
            "~/Desktop"}
    
    
    
    
    -- Write a message into the applescript editor event log.
    log "  --- Starting on " & ((current date) as string) & " --- "
    
    
    log "folderList = " & folderList
    
    set actualCd to do shell script "cd ~;pwd"
    log "actualCd = " & actualCd
    
    
    
    repeat with theFile in folderList
        
        log "theFile = " & theFile
        
        set firstRunning to ""
        
        set expandPath to alterString(theFile, "~", actualCd)
        log "expandPath = " & expandPath
        
        
        debug("SpotlightIndexedData ", expandPath, "SpotlightIndexedFolder")
        
    end repeat
    
    
    
end run

-- ---------------------------------------------
(*
  Create or add to a log file

  based on log by James Reynolds, 12.18.2000, University of Utah
  called recursively.

  input: 
    text string message writen to file
     expanded posix path to file.  ( no ~ in string )
    file name
   
 *)

on debug(theMessage, thePath, theFileName)
    global firstRunning
    local theSize, startupDiskName, fileToLog, fileReference, appPath, changedPerms
    set lineEnd to ASCII character 10
    
    log "==> debug called"
    log "  theMessage = " & theMessage & " class is " & class of theMessage
    
    log "  thePath = " & thePath
    log "  theFileName = " & theFileName
    
    set savePerms to ""
    
    set imbedPath to theFileName & " on  " & alterString(thePath, "/", "-") & ".txt"
    log "imbedPath = " & imbedPath
    
    set unixPathToLog to (thePath & "/" & imbedPath)
    log "unixPathToLog = " & unixPathToLog
    
    
    set fileToLog to POSIX file unixPathToLog as string
    log "fileToLog = " & fileToLog
    
    -- Can we write to this folder?
    
    try
        set fileReference to (open for access file fileToLog ¬
            with write permission)
    on error errorMsg
        log "got error >" & errorMsg & "< "
        
        -- log an attempted permission change on a folder
        set permFileName to "spewChangePermissionList"
        log "permFileName = " & permFileName & " class is " & permFileName
        
        -- check for recursive looping
        if theFileName is equal to permFileName then
            display dialog "This is bad.  Looks like a loop. thePath = " & thePath ¬
                & " theFileName = " & theFileName
            return
        end if
        
        set permFolder to path to home folder
        set unixPermFolder to POSIX path of permFolder
        log "unixPermFolder = " & unixPermFolder
        
        set permList to do shell script "ls -ld " & quoted form of thePath
        log "permList = " & permList & " class is " & class of permList
        
        set priorRunning to firstRunning -- probably "" anyway.
        -- keep track of the folders we plan on changing
        debug(permList, unixPermFolder, permFileName)
        log "... from debug"
        set firstRunning to priorRunning
        
        -- perserve permissions.
        set savePerms to do shell script "stat -f \"%Lp\" " & quoted form of thePath
        
        -- fix permissions if we can.
        try
            set cmd to "chmod  a+wx  " & quoted form of thePath
            log "cmd = " & cmd
            
            set fromUnix to do shell script cmd with administrator privileges
        on error errMsg
            log "chmod error...>" & errMsg & "< on " & thePath
            display dialog "Could not change permissions. >" & errorMsg & ¬
                "< on " & thePath ¬
                giving up after 20
            return
        end try
        set fileReference to (open for access file fileToLog ¬
            with write permission)
        
    end try
    
    log "firstRunning = " & firstRunning
    set theSize to (get eof fileReference)
    log "theSize = " & theSize
    
    if firstRunning = "" then
        
        -- New file to create?
        if theSize is equal to 0 then
            appendToFile(fileReference, "New log created on " & ((current date) as string) & " " & lineEnd)
            
            
            close access fileReference
            try
                -- file must be closed for finder to change creator type.
                tell application "Finder"
                    -- TextWrangler
                    set the creator type of the file fileToLog ¬
                        to "!Rch"
                end tell
                
            on error errorMsg
                display dialog "warning problem with setting creator type. >" & errorMsg & "< results for " & fileToLog giving up after 30
            end try
            
            set fileReference to (open for access file fileToLog ¬
                with write permission)
            
        end if
        appendToFile(fileReference, "   --- debug on " & ((current date) as string) & "   --- " & lineEnd)
        tell current application
            set appPath to path to current application
        end tell
        appendToFile(fileReference, "   --- current AppleScript is " & appPath & " ---" & lineEnd)
        
        set firstRunning to "running"
    end if
    
    
    appendToFile(fileReference, theMessage & lineEnd)
    
    close access fileReference
    
    -- restore directory to what permsissions it had.
    
    if savePerms is not equal to "" then
        try
            set cmd to "chmod  " & savePerms & " " & quoted form of thePath
            log "cmd = " & cmd
            
            set fromUnix to do shell script cmd with administrator privileges
        on error errMsg
            log "chmod error on restoring file ...>" & errMsg & "< on " & thePath
            display dialog "Could not change permissions on attempting retore. >" & errorMsg & ¬
                "< on " & thePath ¬
                giving up after 20
            return
        end try
    end if
    
    
    
end debug

-- ------------------------------------------------------
on alterString(thisText, delim, replacement)
    set resultList to {}
    set {tid, my text item delimiters} to {my text item delimiters, delim}
    try
        set resultList to every text item of thisText
        set text item delimiters to replacement
        set resultString to resultList as string
        set my text item delimiters to tid
    on error
        set my text item delimiters to tid
    end try
    return resultString
end alterString

-- ------------------------------------------------------
-- textToList was found here:
-- http://macscripter.net/viewtopic.php?id=15423

on textToList(thisText, delim)
    set resultList to {}
    set {tid, my text item delimiters} to {my text item delimiters, delim}
    
    try
        set resultList to every text item of thisText
        set my text item delimiters to tid
    on error
        set my text item delimiters to tid
    end try
    return resultList
end textToList

-- ----------------------------------------------------------

on appendToFile(fileId, theData)
    
    local theSize, writeWhere
    log "--> class is " & class of theData
    set theSize to (get eof fileId)
    set writeWhere to theSize + 1 as integer
    write theData to fileId starting at writeWhere as «class utf8»
    
end appendToFile