do shell script "curl etc" not working properly.

I have a script that goes in to my telephone switch and using curl, extracts the log of the calls that happened that week.


Normally this is done by login in via Safari and clicking a button to extract the file to the download folder. When done that way the file works (it is compressed) and the extracted file can be read. The file downloaded is named gespr_dat.csv.g, the file you extract from it is gespr_dat.csv that can than be opened by several applications.


When done via curl the file downloaded is gespr_dat.csv.gz, the file extracted from it becoms gespr_dat.csv.gz.cpgz, and can not be opened but keeps looping when trying to open.



FYI my full script is: do shell script "curl http://192.168.1.190/data_tmp/gespr_dat.csv.gz -s -o /Users/me/Desktop/gespr_dat.csv.gz"


Anybody any idea?

MBP, MM, MBP - 10.6 + Windooz XP on a hard partition

Posted on Oct 10, 2012 2:36 AM

Reply
80 replies

Oct 20, 2012 7:05 AM in response to marshaul

O well the beast has a tail.


The script works beautifully when pasted in to Terminal. However when I try to run it from the user crontab it does not work.


I tested the crontab to see if it is fit or corrupted (/usr/bin/open /Applications/TextEdit.app) and that works fine. So I am stumped as to why it does not run from cron.


here is what I have:




outputfile=//Users/ganesh/Desktop/csv-$(date +%Y-Week%V).gz

cookie=/tmp/cookie.txt

auerswald=192.168.1.190

passwort=123

curl -s --cookie-jar $cookie -o /dev/null -d "LOGIN_NAME=admin" -d "LOGIN_NOW=" -d "LOGIN_PASS=$passwort" http://$auerswald/login_json

curl -s --cookie $cookie -o /dev/null http://$auerswald/page_listgespr_export

curl -s --cookie $cookie --location -o $outputfile http://$auerswald/data_tmp/gespr_dat.csv.gz

curl -s --cookie $cookie -o /dev/null http://$auerswald/appclose

rm $cookie







anybody have a clue as to why?


Message was edited by: ChangeAgent

Oct 20, 2012 3:46 PM in response to ChangeAgent

OK, do a couple of things (these are all going to be commands in the Terminal).


ls -l /pathto/crontab


(ls is the command to list all files and directories found matching the input path, and the -l flag tells it to list in "long" format, which basically gives us more info.)


The first thing you'll see for your crontab file is something like "-rwxr--r--". Post what's there. The next thing you'll see is a number (ignore that), followed by the owner and group. Is your user the owner of the crontab file?






Next, modify your crontab so that it calls the script every 5 minutes (

*/5 * * * *)
, and it saves the standard output/error to files. Do this last bit by appending


>/Users/yourusername/crontab.out 2>/Users/yourusername/crontab.err


to the line which calls your script, modified according to your actual username. Your crontab will look something like:


*/5 * * * * /pathto/yourscript >~username/crontab.out 2>~username/crontab.err



Now, wait 5 minutes, then come back and check for the files (crontab.out and crontab.err). You can open them in textedit easily with the following command (copy and paste this time if you like):


open -a TextEdit.app ~/crontab.*


(I'd just use "less" myself, but hey. Either is a convenient way to view text files with an unusual or incorrect extension.)


Do the files exist? If so, what do they contain?

Oct 21, 2012 3:33 AM in response to marshaul

marshaul wrote:


Next, modify your crontab so that it calls the script every 5 minutes (

*/5 * * * *)
, and it saves the standard output/error to files. Do this last bit by appending


Since my user crontab exist I thought I do part two of your suggestion.


However I do not understand this. If I have the text of my script in a text (.txt) document, how do I call it? is it executable?


Sorry ** I amt not a champion at this.

Oct 21, 2012 4:17 PM in response to ChangeAgent

ChangeAgent wrote:


marshaul wrote:


OK, do a couple of things (these are all going to be commands in the Terminal).


ls -l /pathto/crontab

here is where I get stuck, this bit does not work.


I do however have a user cron tab, I can open it (crontab -l ) and read what is in it.

What happens? It's difficult for me to guess what may be wrong without the terminal output. "No such file or directory", possibly? Be sure to always share the input and output of terminal commands, otherwise folks will just be guessing.





Since my user crontab exist I thought I do part two of your suggestion.


However I do not understand this. If I have the text of my script in a text (.txt) document, how do I call it? is it executable?


I'm not sure I understand the question. I'll answer the best I can, and then I'll ask a question so I might be able to answer better.


On a unix-based operating systems, file extensions don't have any special power. Nothing prevents a process from treating the data in a file differently from how it was intended. For instance, remember I talked a bit about less, the command line pager tool? Well, less is intended to display the contents of files whose data are encoded text. But it will run the routine on files which aren't quite happily; it doesn't really know or care. If you run a command like "less /usr/bin/unzip" you'll see the text interpretation of the binary, which of course is basically unreadable garbage.


File extensions are little more than flags to hint to the operating system, and the user, what programs know how to deal with a certain file type. Some programs run checks to see if the extension of input files matches a list of useable extensions, but this is not strictly necessary. Others might treat a file differently, or do something different to it, depending on its extension; but again this is not necessary. Most of the time, you can change an extension to whatever you want and things will still work, especially in the command-line environment with command-line tools.


For instance, less, as I mentioned, doesn't really give a squat what the extension is; it can be anything, or nothing.


The command-line editor vi (or the more modern and popular vim, for vi iMproved) similarly will open, display, and allow editing of any file type, regardless of extension. What vim will do, however, is take a quick look at the file extension, and use that for syntax highling, say, if a programming language (C or whatever) code file is opened. That's basically all vim uses the file extension for: to add pretty colors to certain text combinations, depending upon the file extension.


And code files themselves are a great example of all this. With basically every language, routines are written into files which are the same as basic .txt files, but have the appropriate extension (.c or whatever), to identify them to the user, to editors like vi, and to compilers which are finicky enough to care. If you write a routine in C in a .txt file, and then change the extension to .c at the last minute, nothing else has changed; just that extension.


To sum all this up very succinctly: there is no inherent relationship between a file's contents and its extension. In many cases it's a good idea to establish and maintain such a relationship (by consisten use of extensions), but it's not inherent. A file is just a bunch of data, and the extension is just part of the name of the file, and neither directly affects the other.


By convention, binary executables on a unix-based system don't have any extension at all.


(Consider the Mac's .app extension, which is a signifier to the OS that the directory is an application. Within that directory tree – specifically at SomeApp.app/Contents/MacOS/ – will be found binary executables (without extensions!), which are the files actually executed by the OS to make it all happen. Everything else is just data files like icons, settings, etc.)


Now, to get to the point: a shell script is just a text file. Consider the file "foo.txt", which contains exactly one line:


echo "Test!"


(The echo command merely sends its argument to STDOUT, which in the case of something run from the command line, is merely the terminal window itself. So this just prints "Test!" in the terminal window.)



There are actually two ways the OS might know that it's a script, either one of which is enough to make it run the script (both are only used so as to avoid a script suddenly not working when it's called differently), and neither one of which is the extension.


For instance, if I run


sh foo.txt


The script works (I see "Test!" in the terminal).


Now, I might rename it "foo.sh" to remind me that it's a shell script. (Or I might rename it foo.zsh because I use z-shell (zsh) and write a lot of my scripts using its syntax rather than the traditional .sh shell syntax.)


Or, I might not. I might simply rename it "foo", and eliminate the extension. This changes nothing.


Now, I might not want to have to type "sh" every time I wish to call my script. I might wish it instead to behave as though it were a binary executable, so I can merely type "/pathto/foo" rather than "sh /pathto/foo" to execute it, or just ./foo (Note: in a unix command line, the sequence ./ as a path means "here, in the present working directory (PWD)", which is why you might see me call a script using ./foo).


There is a bit of metadata (a permissions flag) which tells the OS that a given file is an executable, rather than a simple data file. (Actually, there are three of them, but you can google "unix permissions" if you want to read how that works.) Note that this bit is metadata; it is not stored as part of the file name or extension. To set the executable permission on a file, the command


chmod +x ~/foo


is used (on the Mac, ~ indicates the users home folder, so using ~ in a path is the same as typing /Users/yourusername). Now I can simply run "./foo" to execute my script.


Only problem is, that is only guaranteed to work from within a shell (terminal) environment which understands .sh script syntax. All we've done is tell the OS that the file is executable. We haven't specified how, and it's not binary so it won't execute itself. A shell, like the sh shell or zsh, will look at the file and say "oh, well it's executable, but it's not binary, so it must be a script", so it will begin running the commands in sequence, one line at a time, as though there were simply typed into the terminal.


Now, if the process which attempts to execute the script isn't sh (or an sh-compatible shell like zsh), it might not know how to "execute" a bunch of lines of text (it doesn't know how to interpret those instructions as instructions). To rectify this, the hash-bang (or "shebang) line is used: the first line of the shell script is set to


#!/bin/sh


or


#!/bin/zsh


or


#!/pathto/whatever_shell_i_want_to_use


Now, on a unix-based system, this should be enough to get any dumb enviroment run the script: it looks at the file, sees the hash-bang, and knows it has to start up sh (or whatever) to interpret the script and carry out its instructions.


Notice that at no point in any of that did the extension actually matter at all. I could have left it as ".txt" and nothing would have changed.


(Incidentally, I hope you aren't actually copying and pasting the contents of the script into your terminal window! 😝 Use the cd (change directory) command to get into the directory where the script is – use tab completion to make that easier – and then type ./yourscriptname to execute it. Then, to repeat, simply press the up arrow followed by return. You can browse through the list of previously typed commands by pressing the up arrow repeatedly. Like I said, you should never do anything twice that you can make your computer do!)




I hope that helped. Now, for my question(s):


What is your script called? Where is it? Is it executable (ls -l will tell you this)? And, finally, what are the contents of your crontab file?

Oct 22, 2012 2:32 AM in response to marshaul

wow thank you for taking the time to write this, it helps me understand much better how it all works.


now I have been experimenting, created a test file named 123 that has as text testing in it. The file is in my home folder, or you like user folder.


User uploaded file


It does not work.


Here the Terminal output:


Seva-Server:~ ganesh$ chmod +x ~/123

chmod: /Users/ganesh/123: No such file or directory

Seva-Server:~ ganesh$ ls -l

total 16

-rw-r--r--@ 1 ganesh staff 345 22 Oct 11:17 123.rtf

drwxr-xr-x 4 ganesh staff 136 5 Oct 15:30 Applications

-rwxr-xr-x@ 1 ganesh staff 1120 18 Oct 16:05 Cronscripttelephone.rtf

drwx------+ 21 ganesh staff 714 22 Oct 11:17 Desktop

drwx------ 7 ganesh staff 238 5 Oct 15:30 Documents

drwx------+ 10 ganesh staff 340 5 Oct 15:30 Downloads

drwxr-xr-x 2 ganesh staff 68 6 Jan 2007 Faxes

drwx------ 44 ganesh staff 1496 5 Oct 15:30 Library

drwx------ 3 ganesh staff 102 5 Oct 15:30 Movies

drwx------ 4 ganesh staff 136 5 Oct 15:30 Music

drwx------ 4 ganesh staff 136 29 Jan 2012 Pictures

drwxr-xr-x 4 ganesh staff 136 5 Oct 15:30 Public

drwxr-xr-x 5 ganesh staff 170 3 Jan 2007 Sites

Seva-Server:~ ganesh$ ./123

-bash: ./123: No such file or directory

Seva-Server:~ ganesh$


As you see the 123 file shows up but does not run.


Or have understood you incorrectly?

Oct 22, 2012 8:14 AM in response to ChangeAgent

I'm happy to help. I learned the stuff I know because some people spent a bunch of their time sharing stuff, for free, with me and others on the internet.


Well, in this case it's clear where the problem is.


I should have been more clear about a particular point: many/most command-line type tools don't really care what extensions they see, in a general sense. That being said, while the extension doesn't really do anything other than indicate, it is part of the name of a particular file and it does need to be included in a path when/if it exists.


In this case, your test script – 123.rtf – still has the .rtf extension (probably placed there automatically by textedit), but you're telling chmod to set the executable permission on the file "123", which doesn't exist. My fix:


mv 123.rtf 123


(I always have a terminal window open and therefore I always find it to be faster/easier/fewer keystrokes to do file management tasks there, but that's just me. File management in particular is incredibly powerful on the command line with a very few simple commands and concepts. The following aside might give you some idea why.)

Oct 22, 2012 8:22 AM in response to marshaul

As a mostly irrelevant aside (you can safely ignore this entire post), you can run that chmod command as though the extension is completely irrelevant to the name using a unix tool called globbing. Globbing refers to the use of the asterix (*) as a "wildcard" in unix paths (occasionally in other places too, but I'll stick to paths here). So you could have just said, "oh I forget if it has an extension or what it might have been, too lazy to look it up, so"


chmod +x 123*


This will match all files which have 123 in the first three characters, followed by any number of any other characters. Of course, this may act on other files, in the case you have multiple files that start with "123". You can also glob in the middle of a path. In case you knew the file had the extension .rtf, which you wanted to remove, but you couldn't remember what the name contained beyond "123", you might run:


mv 123*.rtf 123


(In this case the name contained nothing after the 123, but * accounts for that case, too.)


A couple things to note: globbing is processed by the shell, not by the individual commands. This means it works with just about any command line tools which operates on paths. Also, if you were to have numerous files which matched your 123*.rtf glob, say 123a.rtf and 123b.rtf, the shell is going to run (behind the scenes) one command for each match, to wit:


mv 123a.rtf 123

mv 123b.rtf 123


You won't see that happen, but it does. The problem with that case is that you'll be trying to rename two files to one file name, and the first one will be overwritten by the second. So you need to think a little bit before you start wildly issue commands with expansive globs. 🙂 With great power comes great responsibility.


Still, globs are incredibly powerful, right out of the box. I'm not going to try to give any examples; I know for me it just clicked the first time I had to organize a whole bunch of files in multiple directories, by filename (or something similarly basic), and it just clicked, "oh, hey, I can use that unix wildcard thing to put all these .xxx files in one location with a single command."






At the immediate next level of difficulty is using globs in loops (as always, this works the same in scripts or just typed in the command line). For example, in the previous example, if you didn't want to rename both 123a and 123b to the same file:


for file in ./123*.rtf

mv $file $(basename -s .rtf $file)




That would rename 123a.rtf and 123b.rtf to 123a and 123b, respectively. You can use such a command to strip multiple files of their extensions.


By the way if this interests you at all, you can play with this stuff easily by creating a slew of test files. The "touch" command updates a file's last accessed date, or creates an empty file if it doesn't exist. So you can be like


touch foo{1,2,3,4,5}.rtf


And play around with your new dummy files (foo1.rtf, foo2.rtf, etc). And then, later, to clean up:


rm foo*.rtf


But this is now entirely way too far off-topic, and a discussion for another day. 😉 The real point of all this is that the command line's strict adherence to path names isn't really a limitation. It's a slight complication, but it's necessary to having so much power, so precisely directable.

Oct 23, 2012 4:01 AM in response to marshaul

Ok I think I follow.


Here is my Terminal output, nothing seems to happen. The file 123 has text in it (and no longer .rtf in the title), and if I understand you correct this text (testing) should appear in the Terminal window, correct?


Last login: Tue Oct 23 12:48:28 on ttys002

You have mail.

Seva-Server:~ ganesh$ ls -l

total 15

-rwxr-xr-x@ 1 ganesh staff 270 23 Oct 12:49 123

drwxr-xr-x 4 ganesh staff 136 5 Oct 15:30 Applications

drwx------+ 21 ganesh staff 714 22 Oct 11:41 Desktop

drwx------ 7 ganesh staff 238 5 Oct 15:30 Documents

drwx------+ 10 ganesh staff 340 5 Oct 15:30 Downloads

drwxr-xr-x 2 ganesh staff 68 6 Jan 2007 Faxes

drwx------ 44 ganesh staff 1496 5 Oct 15:30 Library

drwx------ 3 ganesh staff 102 5 Oct 15:30 Movies

drwx------ 4 ganesh staff 136 5 Oct 15:30 Music

drwx------ 4 ganesh staff 136 29 Jan 2012 Pictures

drwxr-xr-x 4 ganesh staff 136 5 Oct 15:30 Public

drwxr-xr-x 5 ganesh staff 170 3 Jan 2007 Sites

Seva-Server:~ ganesh$ mv 123 123

Seva-Server:~ ganesh$ mv 123usage: mv [-f | -i | -n] [-v] source target mv [-f | -i | -n] [-v] source ... directory

Seva-Server:~ ganesh$ mv 123.rtf 123mv: 123.rtf: No such file or directorySeva-Server:~ ganesh$ mv 123.rtfusage: mv [-f | -i | -n] [-v] source target

mv [-f | -i | -n] [-v] source ... directory

Seva-Server:~ ganesh$ mv 123*

usage: mv [-f | -i | -n] [-v] source target

mv [-f | -i | -n] [-v] source ... directory

Seva-Server:~ ganesh$

Oct 23, 2012 5:07 AM in response to ChangeAgent

Not quite sure what you're trying to do here, so I guess I'll just explain the two used commands and hopefully you can figure it out from there.


In general, commands in unix take the form


command -flag1 -flag2 argument1 argument2


Where, in principle, there can be any number of flags or arguments. Flags, in general, are used to specify options, or modes, etc, whereas arguments are used to input the actual data (or path thereto) that the programs runs on.


So, with that in mind...




First of all, the ls command. "List directory contents". ls has a number of possible flags; in this case we're giving it the "-l" flag to put it in "long list" mode, which tells it to give us our list in a more data-packed form. If the command ls is typed with no arguments, e.g.


ls


or


ls -l


It simply lists the present working directory. It accepts arguments in the form of paths, and for each argument it presents a list of files at that location: if it's a directory we get a list of its contents; and if it's a file we get a list with exactly one element, that being the file. e.g.


ls ~/123


or


ls ~/Library /Library


(Try both of those if you like.) In this case, I simply wanted to see the permissions on your script, to verify that the executable bit it set. As it happens, I see that it is – when the ls -l command was used, the string -rwxr-xr-x appears before the name it its entry. This character string is an easy way to convey file permissions, for owner, group, and everyone. In this case the presence of the x characters tells me that the file is set to be an executable (it would read something like -rw-r--r-- were the executable permission not set). This brief wiki paragraph explains the syntax:


http://en.wikipedia.org/wiki/Filesystem_permissions#Notation_of_traditional_Unix _permissions





Next, the mv, for move, command. mv has a couple flags it accepts, but we're not using them. mv operates on at exactly two, or more than two, arguments, e.g.


mv argument1 argument2


or


mv argument1 argument2 argument3


In the case of exactly two arguments, it moves (renames) the file found at the path in the first argument to the path found in the second, i.e. "mv foo bar" takes the file called "foo" and renames it to "bar".


In the case of more tha two arguments, the final argument must be a directory, and it moves all the previous arguments into that directory, i.e. "mv foo bar dir/" moves the files "foo" and "bar" into the directory "dir".


I advised the use of mv only as a way to get rid of those pesky extensions. It is not strictly necessary to getting any of this to work.





rm, for remove, is very straightforward, but I'll spend a minute on it since I did tell you to use it a few times. rm immediately and irrevocably deletes the file found at the path specified by its input argument. If there is more than one input argument, it deletes all the files found at the paths specified by those arguments. If an argument is a directory, it throws an error, unless it is given the -r (recursive) flag, which instructs it to recursively go through its directory tree, deleting each file and then finally the empty directory and any subdirectories. (There are several other flags that provide more advanced control over its behavior. You can always get this info by typing "man command", in this case "man rm", which loads the manual in, by the way, our old friend the less pager.)


rm, as with mv, isn't strictly necessary to get any of this to work, outside its use within your script.






Finally, there is the case of executing a script. If a script is not executable, it can be run by executing the shell it was written for with the script as an argument, e.g.


sh ~/myscript.sh


or, perhaps


zsh ~/myscript.zsh


These commands would run the scripts in their respective shells, in what is called a noninteractive session (your terminal prompt is an interactive session), wherein the shell immediately terminates upon running the script.


However, if the script is executable, it is its own command (or at least, it is treated that way from within a compatible shell). It needs no other command to run it; it is self-executing.


Therefore if I run


ls -l ~/myscript.sh


And I see the x in the permissions string, I know that I can simply type


./myscript.sh


to execute it.





(Tangential explanation which may be safely ignored: Remember, the ./ character string is merely the path which means "here, in the PWD".


Executables/commands entered without a path are searched for within your shell environment's $PATH variable.


(If you wish to take a look at your $PATH variable, just type "echo $PATH" in the command line.)


Any command which is not in a directory listed in the $PATH variable must be called using its path, e.g. I can type


./myscript.sh


or


~/myscript.sh (assuming it actually is in my home folder)


But I can't type just


myscript.sh


Whereas, because ls, for instance, is in the directory /bin/, which is listed in the $PATH variable, I can simply type just


ls



I hope that's enough so you can figure out how to do what you were trying to do.

Oct 23, 2012 6:32 AM in response to marshaul

Thank you very helpful! However I still do not understand how I get terminal in the first place and next cron to execute the following script:


outputfile=//Users/ganesh/Desktop/csv-$(date +%Y-Week%V).gz

cookie=/tmp/cookie.txt

auerswald=192.168.1.190

passwort=123

curl -s --cookie-jar $cookie -o /dev/null -d "LOGIN_NAME=admin" -d "LOGIN_NOW=" -d "LOGIN_PASS=$passwort" http://$auerswald/login_json

curl -s --cookie $cookie -o /dev/null http://$auerswald/page_listgespr_export

curl -s --cookie $cookie --location -o $outputfilehttp://$auerswald/data_tmp/gespr_dat.csv.gz

curl -s --cookie $cookie -o /dev/null http://$auerswald/appclose

rm $cookie


remember I can not save it in Cron, so I am trying to get it to run from the file 123 that we made executable.


User uploaded file


So once the file 123 is executable how do I get it to run from Terminal or Cron for that matter?

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

do shell script "curl etc" not working properly.

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.