do shell script "curl etc" not working properly.

I have a script that goes in to my telephone switch and using curl, extracts the log of the calls that happened that week.


Normally this is done by login in via Safari and clicking a button to extract the file to the download folder. When done that way the file works (it is compressed) and the extracted file can be read. The file downloaded is named gespr_dat.csv.g, the file you extract from it is gespr_dat.csv that can than be opened by several applications.


When done via curl the file downloaded is gespr_dat.csv.gz, the file extracted from it becoms gespr_dat.csv.gz.cpgz, and can not be opened but keeps looping when trying to open.



FYI my full script is: do shell script "curl http://192.168.1.190/data_tmp/gespr_dat.csv.gz -s -o /Users/me/Desktop/gespr_dat.csv.gz"


Anybody any idea?

MBP, MM, MBP - 10.6 + Windooz XP on a hard partition

Posted on Oct 10, 2012 2:36 AM

Reply
80 replies

Oct 25, 2012 8:55 AM in response to ChangeAgent

Are you sure cron actually ran the job (you set it to * * * * * or so, so it runs every minute?)?


Add another line to your crontab (but don't delete the one that's already there):


* * * * * echo "I am STDOUT!" >/Users/ganesh/test.out


Confirm, please, that after some short interval the file /Users/ganesh/test.out is created containing "I am STDOUT!", and that there is then no crontab.out in the same directory.


Also, you might check those mail messages (type mail in the command line, then type the number of each message to view it). Perhaps some of them contain useful information.


Considering that we know A: the script itself works, and B: crontab itself works, there's some silly reason why the combination of the two isn't working. Whatever it is, it will probably end up being a "D'oh!" moment.




Edit: By the way, those files (test.out, crontab.out, and crontab.err) won't be executables: just plain old text files. You'll need to actually issue a command to do something with them, e.g.


less ~/test.out

or

open -a TextEdit test.out

or use "open with" on them in the Finder (same result as that second command).


Now, since you got a file not found error (instead of a permission denied error), that's not the problem at the moment. Just a heads up.

Oct 26, 2012 11:01 AM in response to ChangeAgent

OK, so you have a crontab file which has two jobs in it, both scheduled to run at the same time. One job calls the script, and the other simply runs 'echo "I am STDOUT!"'. Both jobs pipe their output stream into files.


After a few minutes, you find that the file ~/test.out exists, but the files ~/crontab.out and ~/crontab.err do not. Correct?


Also, you have at some point run chmod +x on the current script file? Confirm that


ls -l ~/Desktop


shows "-rwxr-xr-x" before the script



Is all this 100% correct so far?





I'd like to verify that crontab will run any script at all.


Copy and paste the following line into your terminal (do it this way so there's no room for mistakes), and press return:



cd && echo "#\!/bin/sh\necho cron\ ran\ me" >| test.sh && chmod +x test.sh


Then add the following line to your crontab using

crontab -e


Again, copy and paste please:

* * * * * /Users/ganesh/test.sh >/Users/ganesh/Desktop/crontabtest.out 2>&1




Confirm that, after some time, the file crontabtest.out is created on your Desktop. What are the contents of this file?









(Also, you might try moving your real script from the desktop into your home folder, changing the crontab to match, and seeing if that works. But do the above first.)

Oct 27, 2012 12:33 AM in response to marshaul

Never noticed this before, but in Terminal Preferences -> Settings -> Shell -> When the shell exits


You can change it to one of the other two options: close the window, or close the window if the shell exited cleanly (i.e. no errors).


That will at least close the terminal window, allowing you to use the "open"-based crontab you had that did work.


Only problem is I'm not sure that the "open"-based cron job will work if the user isn't logged in, being that "open" relies on Finder and all.

Oct 27, 2012 4:19 AM in response to marshaul

marshaul wrote:


OK, so you have a crontab file which has two jobs in it, both scheduled to run at the same time. One job calls the script, and the other simply runs 'echo "I am STDOUT!"'. Both jobs pipe their output stream into files.


After a few minutes, you find that the file ~/test.out exists, but the files ~/crontab.out and ~/crontab.err do not. Correct?

yes




Also, you have at some point run chmod +x on the current script file? Confirm that


ls -l ~/Desktop


yes and it shows:


drwxr-xr-x@ 3 ganesh staff 102 16 Jul 08:33 SafariSaveTelephoneAndSendAndTrash.app


(I note the .app, is that correct? I did not give it that name.)


and in info it shows:


Unix Executable File




I'd like to verify that crontab will run any script at all.


Copy and paste the following line into your terminal (do it this way so there's no room for mistakes), and press return:



cd && echo "#\!/bin/sh\necho cron\ ran\ me" >| test.sh && chmod +x test.sh


Then add the following line to your crontab using

crontab -e


Again, copy and paste please:

* * * * * /Users/ganesh/test.sh >/Users/ganesh/Desktop/crontabtest.out 2>&1




Confirm that, after some time, the file crontabtest.out is created on your Desktop. What are the contents of this file?



it excist and it is empty.

Oct 27, 2012 1:52 PM in response to ChangeAgent

it excist and it is empty.

Odd that it would be empty. The file is created by the act of piping data to it; no data, no file.


At any rate, the .app extension is not correct. When did you script get the .app extension?


I assume you ran the Appify script on it?


Once upon a time, I was unable to get shell scripts to be executable from the Finder, so I needed to make them .app applications. However, and I don't know what's changed (my scripts or Finder?), but Finder seems quite capable of treating a shell script properly as a simple text file with the executable bit. Meaning the advantages of "appicification" now have to do more with icons and data files outside the script itself, I should think. At any rate, there seems to be no need at this point in time to "Appify" a script to make it double-clickable. All the moreso because you're scheduling it anyway, and therefore not using Finder to open it.


It's also possible Finder itself gave it that extension, although that seems odd as well. You'll note from the "d" right at the beginning of the ls -l output that it is a directory. Something definitely did all that. Not sure what.


If the script was, in fact, Appified, and this Appification occurred before this last round of troubleshooting, I would say that's the problem right there.


I can say for certain that sh will not work to execute an Appified script (it's a directory tree now, and sh doesn't have a clue about its structure). Also, if it is executed directly, it will be opened by the Finder, which will then execute its script, and the result should be the same as using open to execute a pure script. Which is also not what we want.





Whether the script was Appified or not, what I would do is the following (I was going to suggest doing this anyway, now I think it's an even better idea):


create a new file in your home folder, in the following manner, so that it has no excess baggage from anywhere (also don't do anything to it except what I say here):



touch ~/SafariSaveTelephoneAndSendAndTrash


(That won't work if there was a file with that name already in the directory. Make sure there isn't.)


Make it executable:


chmod +x ~/SafariSaveTelephoneAndSendAndTrash


Now, copy the script into this file. You can use copy and paste (if so, use pico for the pasting and saving of the new file). Or, if you have a copy of the script in a pure text file (no rtf, no .app business) you could run


cat /pathto/old_script_file >> ~/SafariSaveTelephoneAndSendAndTrash


That will copy the contents of the old file into the new, without copying any of the file metadata as well.




At this point we will have a brand new, unadulterated text file containing your script. Leave it in your home folder for the time being.


Modify your crontab (crontab -e) to:


*/5 * * * * /Users/ganesh/SafariSaveTelephoneAndSendAndTrash >/Users/ganesh/crontab.out 2>/Users/ganesh/crontab.err


(Note: I changed it to run every 5 minutes, in case the script somehow takes longer than a minute to complete, which would bungle things.)


Does it work? (The contents of the .out and .err files can answer this question.)

Oct 28, 2012 1:07 PM in response to ChangeAgent

The output files should have been in your home folder this time, not on the desktop. (>/Users/ganesh/crontab.out) Sorry about that.


It's to be expected that double-clicking it will open it as an App; including opening the terminal window. At some point the Finder has become smart like that.


This should not happen, however, when the file is launched from the command line, e.g.


~/SafariSaveTelephoneAndSendAndTrash


or


sh ~/SafariSaveTelephoneAndSendAndTrash




Does the script run via that second command in the Terminal (not through cron but just entered manually)?


Also, what OS are you running?

Oct 29, 2012 7:19 AM in response to ChangeAgent

OK, well at this point I have no idea what's going on. Without access to your computer I don't really have much idea how to tell you to proceed figuring out what's up with cron. All this works for me.


cron has been deprecated by Apple, in favor of launchd. While cron should (does for me) work the same, maybe this would be a good time to go to launchd. launchd actually offers more configurability than cron, however the plist files are a pain to create.


Use this:


http://launched.zerowidth.com/help


http://launched.zerowidth.com


to generate a launchd.plist. Check out that help one for instructions installing.


Enter the path to your script in the top right field, "Command".

Oct 30, 2012 11:01 PM in response to ChangeAgent

One last thing. Try this; I bet it works:


For your crontab:


* * * * * /bin/sh /Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash


And change your script as follows (changes in red):



outputfile=//Users/ganesh/Desktop/csv-$(/bin/date +%Y-Week%V).gz
cookie=/tmp/cookie.txt
auerswald=192.168.1.190
passwort=123
/usr/bin/curl -s --cookie-jar $cookie -o /dev/null -d "LOGIN_NAME=admin" -d "LOGIN_NOW=" -d "LOGIN_PASS=$passwort" http://$auerswald/login_json
/usr/bin/curl -s --cookie $cookie -o /dev/null http://$auerswald/page_listgespr_export
/usr/bin/curl -s --cookie $cookie --location -o $outputfilehttp://$auerswald/data_tmp/gespr_dat.csv.gz
/usr/bin/curl -s --cookie $cookie -o /dev/null http://$auerswald/appclose
/bin/rm $cookie

Oct 30, 2012 11:12 PM in response to marshaul

Well, the color didn't show up in the code box, and I ran out of edit time, so...


All I did was change everything to an absolute path, both at the crontab level and the shell script level.


sh -> /bin/sh


date -> /bin/date <---- First line of the script, at the beginning of the $() block


curl -> /usr/bin/curl


rm -> /bin/rm


I think that's all of them.



(If need be, do verify that the script, as modified, still works when run directly from the command line.)

Nov 1, 2012 3:31 AM in response to marshaul

marshaul wrote:



(If need be, do verify that the script, as modified, still works when run directly from the command line.)


Sorry but no it does not work, I pasted it direct in to Terminal.



Seva-Server:~ ganesh$

Seva-Server:~ ganesh$ outputfile=//Users/ganesh/Desktop/csv-$(/bin/date +%Y-Week%V).gz

Seva-Server:~ ganesh$ cookie=/tmp/cookie.txt

Seva-Server:~ ganesh$ auerswald=192.168.1.190

Seva-Server:~ ganesh$ passwort=123

Seva-Server:~ ganesh$ /usr/bin/curl -s --cookie-jar $cookie -o /dev/null -d "LOGIN_NAME=admin" -d "LOGIN_NOW=" -d "LOGIN_PASS=$passwort" http://$auerswald/login_json

Seva-Server:~ ganesh$ /usr/bin/curl -s --cookie $cookie -o /dev/null http://$auerswald/page_listgespr_export

Seva-Server:~ ganesh$ /usr/bin/curl -s --cookie $cookie --location -o $outputfilehttp://$auerswald/data_tmp/gespr_dat.csv.gz

curl: no URL specified!

curl: try 'curl --help' or 'curl --manual' for more information

Seva-Server:~ ganesh$ /usr/bin/curl -s --cookie $cookie -o /dev/null http://$auerswald/appclose

Seva-Server:~ ganesh$ /bin/rm $cookie

rm: /tmp/cookie.txt: No such file or directory

Seva-Server:~ ganesh$


Message was edited by: ChangeAgent

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

do shell script "curl etc" not working properly.

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.