Searching for Text in a photo

The following AppleScript will prompt you for the folder containing your screen captures. The image format is not important. It will also prompt you for the word or string of words that you want to match in the OCR'd image text. If there is a match, then the filename will be appended to a text file on the Desktop (matchfile.txt).

Copy and paste the following AppleScript code into Apple's Script Editor, click the hammer icon to compile the script, and then click Run. The code required to output to a scrollable window is double the code shown here and I have opted to write the matching files to a text file instead.

use framework "Foundation"
use framework "Vision"
use AppleScript version "2.4" -- macOS Yosemite 10.10 or later
use scripting additions

property ca : current application
-- this will process all images. If just png then "public.png" or jpg then "public.jpeg"
property screen_shot_uttype : "public.image"
property outFile : POSIX path of ((path to desktop as text) & "matchfiles.txt") as text

set theDir to POSIX path of (choose folder default location (path to desktop)) as text
set the imgStr to (text returned of (display dialog "Enter image text search string: " default answer "")) as text
set matchedFile to ca's NSMutableString's new()

-- a list of all images in the selected folder of type screen_shot_uttype
set imgList to paragraphs of (do shell script "mdfind -onlyin " & theDir & " 'kMDItemContentTypeTree == \"" & screen_shot_uttype & "\"'")
set imgArray to ca's NSArray's arrayWithArray:imgList

-- heading
matchedFile's setString:("Files found that have string: " & imgStr & " in them.")
matchedFile's appendString:(return & return)

repeat with anImage in imgArray
	
	set tildeFileName to ((ca's NSString's stringWithString:(anImage as text))'s stringByAbbreviatingWithTildeInPath()) as text
	set ocrText to my getImageText(anImage)
	-- deal with multi-line text returned as list items and turn it into a return punctuated string
	if class of ocrText is list then
		set ocrText to ((ca's NSArray's arrayWithArray:ocrText)'s componentsJoinedByString:return) as text
	end if
	
	-- do we have a match in the OCR'd image text
	if ((offset of imgStr in ocrText) is greater than 0) = true then
		(matchedFile's appendString:tildeFileName)
		(matchedFile's appendString:return)
	end if
	
end repeat
-- output file
matchedFile's writeToFile:outFile atomically:true encoding:(ca's NSUTF8StringEncoding) |error|:(reference)
return

on getImageText(imagePath)
	-- Get image content
	set theImage to ca's NSImage's alloc()'s initWithContentsOfFile:imagePath
	
	-- Set up request handler using image's raw data
	set requestHandler to ca's VNImageRequestHandler's alloc()'s initWithData:(theImage's TIFFRepresentation()) options:(current application's NSDictionary's alloc()'s init())
	
	-- Initialize text request
	set theRequest to ca's VNRecognizeTextRequest's alloc()'s init()
	
	-- Perform the request and get the results
	requestHandler's performRequests:(ca's NSArray's arrayWithObject:(theRequest)) |error|:(missing value)
	set theResults to theRequest's results()
	
	-- Obtain and return the string values of the results
	set theText to {}
	repeat with observation in theResults
		copy ((first item in (observation's topCandidates:1))'s |string|() as text) to end of theText
	end repeat
	return theText
end getImageText

Let's say I have a screen capture of the following:

In the script's prompt, I enter the following:

and in the Desktop matchfiles.txt there is an entry:

It's actually not possible.

As dialabrain already indicated, that is not possible via the Finder or any similar file finder/viewer.

While certain apps (e.g. Photos) do have some limited capability to find text in an image, the emphasis is on limited. Even Adobe recommends that images (jpeg, png, tiff, etc.) be converted to PDF in order to identify/extract text. (See How to extract text from an image.) Even then, you cannot just "search for all files containing a particular text string."

And accurately searching for text in a PDF can be more complicated, but not impossible either. I recently helped a user who had 4 - 6 digit invoice numbers located in one-page PDFs and I was able to extract those invoice strings and rename the individual PDF to the captured invoice number.

I copied the entire code I posted above (Feb 27, 2024) and pasted it into Script Editor, clicked compile, and ran it against a folder on my Desktop that contained a screen capture image that had macOS Sonoma 14.1.2 in it. The matchfiles.txt file written to my Desktop contained the path to that screen capture image.

No errors were encountered at all but VNRecognizeTextRequest class requires macOS Catalina (10.15+) or later.

Tested: macOS Sonoma 14.4.1.

NHBulldog wrote:

I looked, I don't see where I can save my screenshots as a PDF, so unless there is another app I should look at, I guess this is a "you can't do that." situation.

You can convert images to PDF a number of ways. Preview ... Automator ... Adobe Acrobat ... Acrobat Online Services ... MS Word ... OCR apps ...

Also, you can select files in the Finder, then Control-click the selected files, then choose Quick Actions > Create PDF

You can also change the default file format for screenshots with a pair of terminal commands:

defaults write com.apple.screencapture type pdf

killall SystemUIServer

jpg, tiff, gif, pdf & png are supported file types. png is the standard default.

Thank you for this.

Though it seems to error out with:

error "missing value doesn’t understand the “TIFFRepresentation” message." number -1708 from missing value

and

error "missing value doesn’t understand the “performRequests_error_” message." number -1708 from missing value

You're welcome.