do shell script "curl etc" not working properly.

I have a script that goes in to my telephone switch and using curl, extracts the log of the calls that happened that week.


Normally this is done by login in via Safari and clicking a button to extract the file to the download folder. When done that way the file works (it is compressed) and the extracted file can be read. The file downloaded is named gespr_dat.csv.g, the file you extract from it is gespr_dat.csv that can than be opened by several applications.


When done via curl the file downloaded is gespr_dat.csv.gz, the file extracted from it becoms gespr_dat.csv.gz.cpgz, and can not be opened but keeps looping when trying to open.



FYI my full script is: do shell script "curl http://192.168.1.190/data_tmp/gespr_dat.csv.gz -s -o /Users/me/Desktop/gespr_dat.csv.gz"


Anybody any idea?

MBP, MM, MBP - 10.6 + Windooz XP on a hard partition

Posted on Oct 10, 2012 2:36 AM

Reply
80 replies

Oct 10, 2012 2:57 AM in response to ChangeAgent

Not quite sure what Safari has to do with this...


Why are you using AppleScript to do nothing other than execute a shell script? Kind of missing the point, and redundant to boot.


Instead, simply run the command in Terminal (everything in the quotes). If you need to do it remotely (is that where Safari comes into it?) just use SSH.


No reason to use AppleScript, or Safari, to do what you're doing.

Oct 10, 2012 3:29 AM in response to marshaul

As for the compression loop, I would write an actual shell script (shell scripts are generally named with the extension ".sh").


My shell script might resemble the following:


#!/bin/sh

curl http://192.168.1.190/data_tmp/gespr_dat.csv.gz -s -o /Users/me/Desktop/gespr_dat.csv.gz

gunzip /Users/me/Desktop/gespr_dat.csv.gz

rm -f /Users/me/Desktop/gespr_dat.csv.gz



(The first line declares the shell used to execute the script, the third line unzips it, and the fourth line deletes the original once unzipping is complete. Note that this forum wraps the second line due to its width. It still counts as one line.)


Once you've written and saved your script, run the following command to change the simple text file to an executable script:


chmod +x /pathto/myscript.sh


Then execute it as follows:


/pathto/myscript.sh


Or, you can put it in your home folder and, since Terminal defaults to using your home folder as the present working directory upon launch, you can simply run


./myscript.sh





Note: I suppose if you're deadset on being apple to double-click your script to launch it, AppleScript is an easy way to do that. I'd still keep it a raw shell script, and use something like the Appify script (https://gist.github.com/674099) to make my shell script a double-clickable app.


AppleScript is, IMO, a lot of overhead for something this simple.

Oct 10, 2012 5:49 AM in response to marshaul

marshaul wrote:


Not quite sure what Safari has to do with this...


Safari is how you normally get it, as explained. for my script Safari is not needed. However, when I download it through safari I can unzip it (see original post) when downloading through the script I can not!



Why are you using AppleScript to do nothing other than execute a shell script? Kind of missing the point, and redundant to boot.

Because after this it is send per e-mail to me and next the file is moved to the trash. And I am not good at all at Terminal etc. Furthermore this script is executed by cron.



Instead, simply run the command in Terminal (everything in the quotes). If you need to do it remotely (is that where Safari comes into it?) just use SSH.


not remotely, this happens on my blind server.


Message was edited by: ChangeAgent

Oct 10, 2012 5:53 AM in response to marshaul

Thank you. My problem is not so much if I should use AS or your method, my problem is that when using shell I can not unzip it, while if I download it through Safari I can.

marshaul wrote:


As for the compression loop, I would write an actual shell script (shell scripts are generally named with the extension ".sh").


My shell script might resemble the following:


#!/bin/sh

curl http://192.168.1.190/data_tmp/gespr_dat.csv.gz -s -o /Users/me/Desktop/gespr_dat.csv.gz

gunzip /Users/me/Desktop/gespr_dat.csv.gz

rm -f /Users/me/Desktop/gespr_dat.csv.gz

unfortunately it does not unzip it. I get:


gzip: /Users/me/Desktop/gespr_dat.csv.gz: not in gzip format


If I only run the first line it works fine, but as said in my original post I can not unzip it.


My conclusion is that somewhere this method, like mine, corrupts the file in the process, as a direct download does not corrupt it and allows expanding it in to a readable file.

Oct 11, 2012 9:44 AM in response to ChangeAgent

ChangeAgent wrote:


Using the curl script I get:



/Desktop/gespr_dat.csv.gz: HTML document text




OK,


If you need to be logged, curl can't download the file, the downloaded file should contain a message from the server (4 KO), or it's empty.


if the ÙRL is redirected, you must use the -L option

do shell script "curl -L http://192.168.1.190/data_tmp/gespr_dat.csv.gz -s -o /Users/me/Desktop/gespr_dat.csv.gz"



I can't really help you because I don't have access to the site of the file to download.

But there are certainly more qualified than me to help you.

Oct 11, 2012 1:46 PM in response to Jacques Rioux

I think you might have figured it out here.


ChangeAgent, go ahead and try the last two suggestions (-L and --data-binary).


However, I think the problem is due to login (or lack thereof).


Note that the -o option of the curl command renames the file according to the string which follows (in your case /Users/me/Desktop/gespr_dat.csv.gz).


Now, if the file it's getting is an HTML file, and you're giving it a .gz extension, that's going to confuse archive tools 'til the cows come home.


Give your file a .html extension, and open it up. What happens?




Now, you can use curl to set a login and pass, assuming the webserver in question doesn't require these to be entered in silly fields in a web browser window.


Question: when logging in to download this file via Safari, are you prompted for login via a popup window, or via fields in a normal browser window?



Note: You can easily automate normal shell scripts with cron. Read the following:

http://www.adminschoice.com/crontab-quick-reference


Note also that my script deletes the original file outright (rather than merely moving it to the trash), which would seem to be actually more time-saving in this instance. Anything that can be scripted doesn't need to be saved in the trash. =)


And finally, note that it's very easy to send files via email using the command line (or a shell script). To do this, the sendemail client is probably the best and lightest-weight way to do it.


http://caspian.dotconf.net/menu/Software/SendEmail/


Not necessary to do anything complicated; it's just a perl script, so no need to compile it or anything. Just download it and execute it like you did your shell script. If you want to make it so that you can just type "sendemail" at the prompt (rather than having to type its path first), you can do the following:


sudo mv /pathto/sendemail /usr/bin


(Note that either way you'll probably have to make it executable like you did for your shell script, e.g. "chmod +x /pathto/sendemail". You'll have to use "sudo chmod +x /pathto/sendemail" if you put it into /usr/bin first, due to permissions.)


To send a file using sendemail (you can put this in your shell script, once you get everything else working), check out the following examples:


http://www.debianadmin.com/how-to-sendemail-from-the-command-line-using-a-gmail- account-and-others.html


Shell scripting is a good way to do these kind of automated tasks, as you're ensuring the absolute minimum amount of overhead (which is always a good thing for automated tasks which are executed while you might be doing other things).

Oct 11, 2012 11:15 PM in response to Jacques Rioux

good morning,


Jacques Rioux wrote:


OK,


If you need to be logged, curl can't download the file, the downloaded file should contain a message from the server (4 KO), or it's empty.

as I can not open it I can not see what is inside.




if the ÙRL is redirected, you must use the -L option


do shell script "curl -L http://192.168.1.190/data_tmp/gespr_dat.csv.gz -s -o /Users/me/Desktop/gespr_dat.csv.gz" 



no difference, the resulting file does not open.

Oct 11, 2012 11:48 PM in response to marshaul

Thank you for this very interesting and instructive information.

marshaul wrote:


I think you might have figured it out here.


ChangeAgent, go ahead and try the last two suggestions (-L and --data-binary).



see the other answers, I did not solve it.


And yes my suspicion is too that is it due to needing to log-in.




Note that the -o option of the curl command renames the file according to the string which follows (in your case /Users/me/Desktop/gespr_dat.csv.gz).


Now, if the file it's getting is an HTML file, and you're giving it a .gz extension, that's going to confuse archive tools 'til the cows come home.



yes I know about the 'o' option. However the name we give to the document is the name it gets if I log in and use the download button. The resulting document is indeed a compressed file. If I open it I get a file that can be read in, for example, TextEdit. The reulting TextEdit document is called: gespr_dat.csv




Give your file a .html extension, and open it up. What happens?

had already tried that before, no cigar. If I do I get:


Access Error: Forbidden

Bad state


Now, you can use curl to set a login and pass, assuming the webserver in question doesn't require these to be entered in silly fields in a web browser window.


Question: when logging in to download this file via Safari, are you prompted for login via a popup window, or via fields in a normal browser window?


Yes and as a picture tells more than a thousand words...


The log In window is a pop-up. (sorry interface only available in German, the price you pay if you want good old German quality equipment :-) )


User uploaded file


the download than happens via a button.


The URL next briefly changes, browser window does not change, so I see no new window.


URL is:


User uploaded file


temporary (2 seconds) URL is (as the downloaded file appears on the DT).


User uploaded file


Message was edited by: ChangeAgent

Oct 12, 2012 12:00 AM in response to ChangeAgent

OK, well by popup I really meant the window built-in to Safari which prompts for login/pass (you'll see this on some members-only websites and the like).


That window is indeed a popup, but it's one generated by the device's HTML server, which isn't what I was hoping for. Sorry for being vague. Your picture was worth a thousand words, however.


Try using less (as I suggested in my post immediately prior to this one), and see what the contents of the file are.


However, this is only to confirm that the nonworking file is, in fact, an HTML file.


At this point, however, I'm fairly certain that it is, and it's because it wants login credentials. Unfortunately, thanks to their using an HTML form, this will be a little more difficult than merely passing login credentials directly via curl.


But wait! All is not lost! You can still resolve this issue, with a bit more work.


Take a look at the following video:


http://www.linuxjournal.com/video/use-curl-login-websites-script


This tells you how to use curl to submit login credentials passed by an HTML form (curl is awesome, really).


Once you've figured it out, you can put all the commands in a shell script, use crontab to automate it, and never look back. =)


(This is why it really pays to learn how to use the command line, although I know that's not a trivial task – but hey, you're starting right now! =). You can really make your computer work for you, and automate everything to boot. The environment is so powerful, that if a task is possible for a computer to perform, you can script it. I never do repetitive tasks manually – that's what computers are FOR!)

Oct 11, 2012 11:58 PM in response to marshaul

marshaul wrote:


Sure you can. Either change the extension (.html is your best best bet because that's what the file is reported to be), or just enter the following in the terminal:


</Users/me/Desktop/gespr_dat.csv.gz


(Press 'q' to exit the less pager when you're done.)

as said, HTML I get


Access Error Forbidden.


your code with the < does do nothing, without the < gives me


Permission Denied

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

do shell script "curl etc" not working properly.

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.