Making AppleScript Curl FTP transfers faster.

Question

Level 1

0 points

Making AppleScript Curl FTP transfers faster.

I am using Applescript to transfer files from our local Mac OS web server to one at a different location. I am using statements like this:

tell application "Finder" to do shell script "curl -T /Users/FILE/PATH/HERE/filename.jpg ftp://username:password@domain.com/some/path/filename.jpg"

This works great. It is reliable. The issue is this. We are transferring thousands of files like this all day long, and it must be fast -- as fast as possible.

Right now it takes about 4 seconds each time to transfer even tiny files. I am guessing a major part of this time is spent opening and closing the FTP connection.

Is there some way to "permanently" keep alive the FTP connection curl is using to do this, so anytime something is uploaded to our FTP server via AppleScript/curl, the connection will already be active? Then, maybe uploads will be much faster.

If you have a suggestion how to do this, complimenting the method we are already using, or a suggestion of a better way to do this altogether, I'd enjoy learning it. Thank you.

iBook, Mac OS X (10.4.11)

Posted on Sep 21, 2010 3:14 PM

Reply

Answer 1

Sep 21, 2010 3:56 PM in response to QuakingBog

Hi,

Take a look at: http://curl.haxx.se/docs/manpage.html and the -T flag.

You can specify one -T for each URL on the command line. Each -T + URL pair specifies what to upload and to where. curl also supports "globbing" of the -T argument, meaning that you can upload multiple files to a single URL by using the same URL globbing style supported in the URL, like this:

curl -T "{file1,file2}" http://www.uploadtothissite.com

or even

curl -T "img[1-1000].png" ftp://ftp.picturemania.com/upload/

Note the destination directory should end in a trailing slash using this method (otherwise it might replace the directory with a file). So you could probably create a list of filepaths and then add them as text to the command. There is a maximum length you can make a 'do shell script' command. See: http://developer.apple.com/library/mac/#technotes/tn2002/tn2065.html

Also, unless there is more to the script, there is no need to have the +tell application "Finder"+ part.

Best wishes

John M

Reply

Answer 2

QuakingBog Author

Level 1

0 points

Sep 21, 2010 6:04 PM in response to John Maisey

Thanks for the suggestions, John. By grouping some of the transfers together, that should speed things up a bit. I will do that.

What I am also interested in is knowing if there is a way to keep the connection "alive" in between times that CURL is being called from AppleScript? (Is that what is known as KeepAlive?) In other words, a way to keep the FTP connection always active, so the transfers themselves go a bit faster. I don't know technically what is going on "behind the scenes" but I'll illustrate with an example from another program I use regularly, BBEdit.

If I used BBEdit's FTP function to connect to a remote server, it initially takes several seconds to connect, but once connected the selected file opens and save instantly, fractions of a second. Then, if I open another or save another file, it also opens instantly, as long as BBEdit is still connected to that remote server.

If BBEdit closes the connection to the server, then the next time I am go to open a file, I will need to wait the initial several seconds for the FTP connection, then the transfers go quickly again.

In my own case, I am using Applescript with Curl to transfer JPEG files. The files are transferred thousands of times per day, triggered by actions on our website. The only thing is, there might be a delay of a few seconds, or even several minutes, between files needing to be uploaded.

What I am trying to eliminate is the "ramp up" delay which occurs each time Curl/AppleScript initiates the FTP connection. In other words, I am trying to find a way to create and maintain a permanent FTP connection, even if no transfers are going on at that particular instant.

The script runs, then ends. Then later a new script runs again, so I'm not sure if there is a way to keep the connection active between the times each script runs.

Is this something which can be done with CURL/AppleScript, or will I need to use some sort of scriptable FTP client to do this? Any suggestions would be very appreciated!

Reply

Answer 3

Camelot

Level 9

54,333 points

Sep 21, 2010 9:44 PM in response to QuakingBog

Each invocation of do shell script spawns a whole new process, so there is no association between one and another, so what you describe can't be done.

The aforementioned techniques to tell curl to make multiple transfers is the easiest way to do what you describe (you invoke curl once and tell it to transfer multiple files before it quits).

Other things you can do to help include:

1) drop the 'tell application "Finder" element. There is no requirement (or even desire) to involve the Finder in this process.

2) If network bandwidth isn't an issue, run multiple curl processes in parallel, e.g.:

do shell script "curl -T /Users/PATH/file1.jpg ftp://user:pass@server.dom/path/file1.jpg > /dev/null 2>&1 &"
do shell script "curl -T /Users/PATH/file2.jpg ftp://user:pass@server.dom/path/file2.jpg > /dev/null 2>&1 &"
do shell script "curl -T /Users/PATH/file3.jpg ftp://user:pass@server.dom/path/file3.jpg > /dev/null 2>&1 &"

Appending '> /dev/null 2>&1 &' to the end of the shell command tells it to run in the background and allows your script to continue (normally it will wait until the shell command finishes before continuing). In this way you can spawn multiple curl instances.

3) If the above doesn't work, switch to some other FTP client that has a decent AppleScript interface.

4) Switch to some other protocol that has better OS integration/state mainenance. e.g. if you can use AFP to mount the remote server's volume on your desktop then you can use regular file copy commands to upload your files.

Reply

Answer 4

QuakingBog Author

Level 1

0 points

Sep 22, 2010 6:47 AM in response to QuakingBog

Thanks for the additional great suggestions. I've tried a few more experiments and found that using the following AppleScript works almost twice as fast as using CURL:

tell application "URL Access Scripting"
upload filepathtoUpload to "ftp://username:password@domain.com/SOME/PATH/filename.jpg" replacing yes without binhexing
end tell

Later today I will try using AFP as you suggested, and compare that to the speed of URL Access Scripting.

Reply