Applescript/Automator - Open all links on a webpage

Question

Applescript/Automator - Open all links on a webpage

I am stuck trying to build an Applescript or Automator Workflow that goes to a predefined web page, and then loads each of the links found on the web page. The catch is that I want to loop through one at a time (with a delay between each one).

I've been able to get all the urls to load simultaneously via Automator, but can't figure out how to loop through the urls with a delay.

Thank you in advance for any tips.

Applescript-OTHER, Mac OS X (10.7.2)

Posted on Nov 2, 2011 11:33 AM

Reply

Answer 1

Best reply

Frank Caggiano

Level 7

25,899 points

Nov 2, 2011 6:58 PM in response to karlfromnorfolk

This gets the URL's from the Specified page and then passes them to an AppleScript that opens each one in turn waiting after opening each for you to press continue.

It's really barebones, there is no error checking and there are probably better ways to handle opening the pages in Safari but I think it gives an idea of how to do what you are looking for.

regards

Just noticed in you post you said you wanted to delay between pages so I took out the user input to continue and added a delay statement. Right now it delays for 10 secs. You can of course change that.

Message was edited by: Frank Caggiano

Reply

Answer 2

Nov 2, 2011 7:20 PM in response to karlfromnorfolk

Thank you so much. That is exactly what I needed.

Reply

Answer 3

Nov 2, 2011 7:32 PM in response to karlfromnorfolk

Glad it helped.

Just realized that having the tell application "Safari" inside the URL loop is wasteful. Probably not a big deal here but you might want to change the Applescript part to:

on run {input, parameters}

tell application "Safari"

activate

repeat with i from 1 to length of input

set the URL of front document to item i of input

delay 10

end repeat

end tell

return input

end run

good luck

Reply

Answer 4

Nov 3, 2011 6:42 AM in response to Frank Caggiano

Oh that's perfect. That allows me to run this and have it loop through the pages in the background without constantly yanking focus back to the safari window. Thank you so again for taking the time to help me on this.

Reply

Answer 5

Nov 4, 2011 11:54 AM in response to Frank Caggiano

Frank. I have one additional tweak I'd like to make. After the URL loads in the SET URL step of the loop, I would like to check for the presence of a certain url in the body of the page. Say if there is a url starts withhttp://www.google.com -- I would like to follow that URL. It can just follow the first one it finds (or all of them).

Reply

Answer 6

Nov 4, 2011 1:33 PM in response to karlfromnorfolk

Well that adds quite a wrinkle! Off the top of my head I can;t think of any easy way to do that. The web page will have to be parsed looking for URLs And when you say you want to 'follow' it how deep do you plan to go? If the new page has the URL you're looking for do you then follow it?

The whole script may need to be redone in this case scraping the Automator part and doing it all in AppleScript or shell scripting.

Let me think on this in the meantime maybe someone else will come up with a simple way I'm over looking.

regards

Reply

Answer 7

Nov 4, 2011 1:39 PM in response to Frank Caggiano

It just needs to go one layer deep.

I was wondering if there was a way to stick an automator get urls and filter urls object inside the loop that we have working. I tried it and it automator didn't like that.

Reply

Answer 8

Nov 4, 2011 1:47 PM in response to karlfromnorfolk

There is a way to run an Automator Action from an AppleScript I've done it in the past but can remember the exact method. I believe you have to resort to actually calling the action via a shell command in the AppleScript.

I don;t have time to look it up right now but if you Google something like automator action in applescript you should get some ideas.

If I have time over the weekend I'll give it a look.

good luck

Reply

Answer 9

Nov 5, 2011 3:11 PM in response to Frank Caggiano

Yes, I found this:

http://hints.macworld.com/article.php?story=20081109152027513

And I created an automator workflow that looks like this:

My thought was that I would call it right after the delay 10 step. Except I can't figure out the syntax of the code to call it. I tried:

set {tid, AppleScript's text item delimiters} to {AppleScript's text item delimiters, linefeed}

set theInputsList to inputVars as text

set AppleScript's text item delimiters to tid

set cmd to "automator -i '" & currentURL & "' " & /Documents/Path/To/Automator Workflows/t-mobile click.workflow

return do shell script cmd

end run_workflow

It doesn't seem to like the syntax of my path.

Reply

Answer 10

Nov 8, 2011 9:26 AM in response to karlfromnorfolk

I figured out how to accomplish this by having applescript fire some javascript code inside safari (I read about it here:

--http://stackoverflow.com/questions/5135609/can-applescript-access-browser-tabs-a nd-execute-javascript-in-them)

. Here's what my applescript looks like now:

on run {input, parameters}

tell application "Safari"

activate

repeat with i from 1 to length of input

makenewdocumentat the front of documents

set the URL of the front document to item i of input

delay 10

do JavaScript "var links = document.getElementsByTagName('a');

var match = 'http://www.google.com';

for(var i=0; i<links.length; i++) {

var href = links[i].getAttribute('href');

if(href.indexOf(match) != -1) {

document.location.href = href;

break;

}

}" in the front document

delay 10

close every window

end repeat

end tell

-- return input

end run

Reply

Answer 11

Nov 8, 2011 12:55 PM in response to karlfromnorfolk

Sorry I wasn't able to get back to this sooner. You've been busy 😉

You're last solution wouldn't have been soemthig that would have poped into my head, but if it works and does what you need great.

I'll have to keep this in mind,

regards

Reply

Answer 12

Nov 11, 2011 7:56 AM in response to Frank Caggiano

Now I'm trying to use the "Filter URLs" action in my worfkflow. However, I cannot for the life of me figure out how to only return urls that DON'T contain a certain string. In automator, I see under "Entire URL" the option of "Contains", "Begins With", Ends With, Is and Is Not.

How do I filter to exclude URLs that contain a certain string?

Reply

Answer 13

Nov 12, 2011 9:17 AM in response to karlfromnorfolk

Just saw this,

I think you might be leaving the realm of what is possible using the methods you used to get to this point. You started out wanting to open all URL's on a page with a time delay between pages, then you wanted to see if a certain link was on the page and it it was follow that and now you want to filter the URL's for certain strings!

What a long strange trip its been (to borrow a quote)!

If I were you I would scrap all this and start over working only in AppleScript and possibly shell scripting. This is common in software development. By starting over fresh you'll take the knowledge you've gained while prototyping and come up with something cleaner in the end.

Had you laid out all the requirements when you first posted I probably would have recommended doing it this way from the start. As a start I would use curl called by using AppleScripts do shell command and then parse the returned web content with Applescript looking for URL's. In AppleScript it will be easy to parse the URL however you wish.

One thing isn't clear to me now that your project has changed was the display of the URL in Safari menat as a means to see the page or only as a way to get the page's content? I think you were only going for the content. If so curl will do nicely.

good luck post back if you need help with the new script. Might be best to start a new thread describing what you've done so far and explaining what you want to do if you need help.

Reply

Answer 14

May 24, 2015 10:28 AM in response to karlfromnorfolk

Hello there,

I have the same problem: I am trying to download all the material from my class site. The class site though is in the a container called "Moodle" that seems to mess up everything. I think I know how to download the urls found in a web page, the problem is that for some reason I do not seem to get the right action to get the url in this page. If I use the " get link Urls from Webpage , it gets the link of the containers (Moodle), but not the one I am interested in. However, if I save the webpage and right click on the link....I can download the material am interested in. Is there a way automator can parse the urls int he page in the way that I can get all the material Iperlinked to the class at once? I apologize if the jargon is confusing, but I am just a novice at any automatization process. Cheers and thanks,

Reply