Web scraping on mac
I am trying to pull URLs from web searches into Excel for the Mac using a macro. I have been able to pull in the search page and store it on a spreadsheet but it retrieves the page as if it took a screenshot, which truncates a lot of information (including the URLs). The ultimate intent is to automate a Google search on a topic by using the macro to retrieve resulting URLs, open those pages, search for keywords & store any hits on a spreadsheet. Once I have a URL (say of a news article), I have been able to copy the text on a spreadsheet, read it into a text string and use the InStr function to search for the keywords. The problem is getting the (intact) URLs from the Google search. There must be a simple way to accomplish this.
The macro includes the following:
sURL="http://www.google.com/search?q=" & <search term>
With ActiveSheet.QueryTables.Add(Connection:="URL;" & sURL, Destination:=Range("$A$1"))
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingAll
.WebPreFormattedTextToColumns = False
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = True
.Refresh BackgroundQuery:=False
End With
If WebFormatting is set to None, it doesn't pull anything in.