do shell script "curl etc" not working properly.

I have a script that goes in to my telephone switch and using curl, extracts the log of the calls that happened that week.


Normally this is done by login in via Safari and clicking a button to extract the file to the download folder. When done that way the file works (it is compressed) and the extracted file can be read. The file downloaded is named gespr_dat.csv.g, the file you extract from it is gespr_dat.csv that can than be opened by several applications.


When done via curl the file downloaded is gespr_dat.csv.gz, the file extracted from it becoms gespr_dat.csv.gz.cpgz, and can not be opened but keeps looping when trying to open.



FYI my full script is: do shell script "curl http://192.168.1.190/data_tmp/gespr_dat.csv.gz -s -o /Users/me/Desktop/gespr_dat.csv.gz"


Anybody any idea?

MBP, MM, MBP - 10.6 + Windooz XP on a hard partition

Posted on Oct 10, 2012 2:36 AM

Reply
80 replies

Oct 23, 2012 1:43 PM in response to marshaul

Once you figure that out, you can make your crontab (test setup for now).


I believe the default editor is vim, which is the best text editor ever written, however it comes with a steep learning curve. I know the first time I tried to edit a file using it I about tore my hair out thanks to its confusing modular nature (it has different modes for entering text vs commands, unlike "normal" editors which basically have one mode, insert text mode), before I switched to another editor.


So, you can either Google "basic VIM usage", or you can type the following command:


export EDITOR=pico


Pico is self-explanatory, though not a fraction as powerful as VIM. Good enough for our purposes, though.


Now you can type


crontab -e


From here, enter this single line into the file, save it and quit (I believe this will be correct, but you'll want to try to understand it before you type it in, in case I made a minor mistake somewhere):



* * * * * /Users/ganesh/123 >/Users/ganesh/crontab.out 2>/Users/ganesh/crontab.err



To understand this line:


* * * * *: This first bit's going to run the file every minute (probably won't happen the first time until the second minute, however).

/Users/ganesh/123: both the path to, and the command to call your script (see my last two posts).


>/Users/ganesh/crontab.out: saves the output stream to a file

2>/Users/ganesh/crontab.err: saves the error stream to another file


The > character, at the end of a unix command, tells the shell to redirect STDOUT (the output "stream" which normally appears as simple text in your terminal window after entering a command) to the file specified after it, in this case ~/crontab.out. Similarly, 2> does the same with STDERR (which is the error "stream", also normally appears in the terminal window) and the file crontab.err.


Now, wait a few minutes, and look at thoe files. What do they contain? (copy and paste the following if you like:)


open -a TextEdit.app ~/crontab.{out,err}

Oct 23, 2012 2:03 PM in response to marshaul

marshaul wrote:


~/123



(I know my posts inevitably end up looking (are?) very long... But I did explain this in detail at the end of my last post.)

yes you did, but here is the output:


Seva-Server:~ ganesh$ ~/123

/Users/ganesh/123: line 1: {rtf1ansiansicpg1252cocoartf1038cocoasubrtf360: command not found

/Users/ganesh/123: line 2: syntax error near unexpected token `}'

/Users/ganesh/123: line 2: `{\fonttbl\f0\fnil\fcharset0 Verdana;}'

Seva-Server:~ ganesh$


it does not do anything that I can recognize.

Oct 23, 2012 2:15 PM in response to ChangeAgent

Sorry, that one's totally my bad. I don't play with rtf files enough for it to have occurred to me like it should have.


See that rtf in the first line? Then the Verdana font tag? Yeah.


Your file is still encoded as an rtf file (remember how I said that extension didn't actually have any effect on the file contents? Yeah. lol. There you go.)


Open the file in TextEdit, and go to Format -> Make Plain Text. Save it.


That ought to do it. You might want to look at the script using less (less ~/123) to make sure that removed all the tags properly. But my guess is it worked fine.

Oct 24, 2012 10:03 AM in response to ChangeAgent

OK. Well, the cron daemon (crond) is always running; otherwise cron doesn't work/is disabled.


That said, the daemon is a background process, so it should be quite invisible to your normal GUI environment. (Not to mention it's lightweight so it shouldn't be hogging resources either.) Any scheduler is going to have to run a daemon like this, incidentally.


So, I have a feeling that's not what you're talking about.


Perhaps you can elaborate on "open cron"? Does it leave an icon in your dock? Or is this something you see in top or activity monitor? Tell me, specifically and exactly, what you're seeing here.


In principle it's no biggie to write a script which kills a process based on name, but that shouldn't have to be done here. I'm not sure what's going on.

Oct 25, 2012 12:47 AM in response to marshaul

marshaul wrote:


Odd. It shouldn't do that, or need to, as far as I can think.


What's your crontab command?


/usr/bin/open /etc





Here is the output ( as you can see the file has been renamed):



Last login: Thu Oct 25 09:43:57 on ttys001

/Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash ; exit;

You have mail.

Seva-Server:~ ganesh$ /Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash ; exit;

Oct 25 09:45:05 seva-server sendEmail[1799]: Email was sent successfully!

logout



[Process completed]

Oct 25, 2012 1:53 AM in response to ChangeAgent

/usr/bin/open /etc


Wait, so if you type


crontab -e


You see something like:


* * * * * /usr/bin/open /etc


???


What that command there does is tell the Finder to open /etc.


open is a non-standard, Mac-centric command, which basically says "Perform the equivalent of the Finder command-o (open) on the target file". So, in this case it will bring up a Finder window at /etc.


This command is unusal in that it depends upon a higher level of arbitration than the command line – i.e. it directly calls the Finder to operate on its input, taking advantage of the Finder's built in file onwership associations, so that if you type "open SomeFile.txt" Finder will say, "oh, TextEdit is associated with .txt files, so I'll open this in TextEdit." This is unusual in that command line routines are – generally – unlikely to depend on such a high level of arbitration (in fact, the Finder itself performs many of its functions using what are essentially shell commands, that being the usual flow of arbitration: from high to low).
open -a "SomeApp.app" SomeFile.txt


Is equivalent to using the Finder "open with" command on SomeFile.txt, and selecting "SomeApp".





So, this generally isn't the kind of task you schedule (it doesn't really do anything). Where did this line come from? Again, is it reflected when you type


crontab -e


?



Also, this line doesn't in any way call or execute a shell script. If that is the contents of your cron job, then the script is being called somewhere else.






As to the script output I see there (which, again, you say is being executed by a scheduler? It's not the result of a manual execution of the script?): that doesn't look like a cron call.


See that "exit" command? That's the command used to logout of an interactive session. And, I daresay, that's why Terminal.app is being launched. The sole difference between an interactive session and a noninteractive session is that an interactive session allows for user input, which requires a Terminal window, hence Terminal.app must be open to receive user output (and also display STDOUT in the Terminal window.


Go ahead: launch Terminal.app, and type exit on a single line, followed by return. See what happens.


exit is therefore redundant in a noninteractive session, which automatically terminates the process once the script execution is complete. It doesn't need an explicit exit command to do so.


So, where is that coming from? Who or what is running the command sequence "/Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash ; exit;" ?




It seems to me like there's some confusion in how this script is being executed, or where, or by what.

Oct 25, 2012 2:07 AM in response to marshaul

On a somewhat unrelated note, the reason your shell session is informing you that you have mail is that cron, by default, emails whatever goes through the STDOUT stream to the user to whom the job belongs.


I'd be curious to see the contents of some of this mail.


You can check this mail by typing


mail


in the command line. Then, to view a message, just type its number, followed by return.


(You can delete mail by typing "d " (don't forget the space!) followed by either the message number, or the group if you wish to empty the whole group. The group is the letter next to the message's number.)




If, in the future, you want to prevent your cron job from sending you these emails (after all, you already send yourself an email upon completion, and one of your own design and intent to boot), you need to redirect STDOUT and STDERR to either a file, if you wish to save logs of these outputs, or to /dev/null, if you don't.


http://en.wikipedia.org/wiki//dev/null


In Unix-like operating systems, /dev/null or the null device is a special file that discards all data written to it (but reports that the write operation succeeded) and provides no data to any process that reads from it (yielding EOF immediately).

In programmer jargon, especially Unix jargon, it may also be called the bit bucket[2] or black hole.


The null device is typically used for disposing of unwanted outputstreams of a process, or as a convenient empty file for input streams. This is usually done byredirection.


This is done by appending


>/dev/null 2>&1


to the crontab/job. (The first part pipes STDOUT to /dev/null, and the second bit pipes STDERR to STDOUT, which of course is already going to /dev/null.)



So you'd have something like:


* * * * * /Users/ganesh/SaveTelephoneAndSendAndTrash >/dev/null 2>&1


when you type crontab -l.



(Don't do that until everything works to your satisfaction. Logs may be useful until then.)

Oct 25, 2012 3:16 AM in response to marshaul

marshaul wrote:


/usr/bin/open /etc


Wait, so if you type


crontab -e


You see something like:


* * * * * /usr/bin/open /etc


no, no, my unclear. I see the command that opens the script we made.


40 20 * * 7 /usr/bin/open /Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash




So, where is that coming from? Who or what is running the command sequence "/Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash ; exit;" ?



It seems to me like there's some confusion in how this script is being executed, or where, or by what.

it is in the crontab that executes the file we called earlier 123.

Oct 25, 2012 3:55 AM in response to ChangeAgent

40 20 * * 7 /usr/bin/open /Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash


Well, there you have it.


/usr/bin/open uses the Finder to open the script, exactly as though you had double clicked it.


(Go ahead, double-click your script in the Finder; see what happens.)


When Finder spawns a Terminal window, it does so in an interactive session. This is because the Finder is a GUI environment, and in a GUI the user generally expects to see feedback when he double-clicks a file (Apple, in particular, is quite consistent on this point). So, scripts run manually from the Finder are executed in an interactive session, so that the STDOUT and STDERR streams can be viewed by the user. Once it's done, the exit command is given to terminate the session.


When a script is executed directly, it is run in a non-interactive session, which terminates of its own accord when it's run through all its instructions. No need to show STDOUT in the GUI. No need for an explicit exit command.


The open command has great utility, when you want to use the command line to open a document in a GUI app, or you want to take advantage of the Finder's file extension associations by opening a document in whatever the default owner of such documents is. That's about the extent of it.


In this case, there's no need to use Finder to invoke your script. In fact, it's the source of the undesired behavior.


So, in your crontab entry, delete



/usr/bin/open



leaving you with just



40      20      *       *       7       /Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash







(If for some reason that doesn't work, you can replace

/usr/bin/open

with

/bin/sh


To explicitly call the shell to invoke the script.


But that really shouldn't be necessary.)

Oct 25, 2012 6:58 AM in response to marshaul


So, in your crontab entry, delete



/usr/bin/open



leaving you with just



40      20      *       *       7       /Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash







(If for some reason that doesn't work, you can replace

/usr/bin/open

with

/bin/sh


To explicitly call the shell to invoke the script.


I tried you first suggestion in te first place (before posting), and than it does not run, so I added

/usr/bin/open
to solve it.


also /bin/sh does not work either.

Oct 25, 2012 7:52 AM in response to ChangeAgent

Couple of things:


That cron job: are you intending to run it once per week? I don't believe 7 is a valid day; pretty sure it's 0 (Sunday) - 6 (Saturday). Also I assume that you're testing it with a more frequent schedule.





More pertinently: you say it doesn't work, but it has to be doing something. Otherwise it wouldn't work either with the added /usr/bin/open.


Just to be sure: you are quite certain it's not working? Keep in mind it won't give you any sort of visual feedback when it runs. Nothing will bounce in your dock. You'll get the email and that's it.


Assuming you are certain, make your cron job like (be sure to scroll to the right):



*      *      *       *       *       /Users/ganesh/Desktop/CronCommands/SaveTelephoneAndSendAndTrash >/Users/ganesh/crontab.out 2>/Users/ganesh/crontab.err



Wait until it runs, then take a look at those files (~/crontab.out and ~/crontab.err). Post them here. That's the only way to know what it's doing.

Oct 25, 2012 8:01 AM in response to marshaul

marshaul wrote:


Couple of things:


That cron job: are you intending to run it once per week? I don't believe 7 is a valid day; pretty sure it's 0 (Sunday) - 6 (Saturday). Also I assume that you're testing it with a more frequent schedule.

I once read that 0 and 7 are Sundays. I change it to 0 but that is not the problem.




Wait until it runs, then take a look at those files (~/crontab.out and ~/crontab.err). Post them here. That's the only way to know what it's doing.


Here is what I get:


Last login: Thu Oct 25 16:59:51 on ttys001

You have mail.

Seva-Server:~ ganesh$ ~/crontab.out

-bash: /Users/ganesh/crontab.out: No such file or directory

Seva-Server:~ ganesh$ ~/crontab.err

-bash: /Users/ganesh/crontab.err: No such file or directory

Seva-Server:~ ganesh$

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

do shell script "curl etc" not working properly.

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.