Level 2

301 points

do shell script "curl etc" not working properly.

I have a script that goes in to my telephone switch and using curl, extracts the log of the calls that happened that week.

Normally this is done by login in via Safari and clicking a button to extract the file to the download folder. When done that way the file works (it is compressed) and the extracted file can be read. The file downloaded is named gespr_dat.csv.g, the file you extract from it is gespr_dat.csv that can than be opened by several applications.

When done via curl the file downloaded is gespr_dat.csv.gz, the file extracted from it becoms gespr_dat.csv.gz.cpgz, and can not be opened but keeps looping when trying to open.

FYI my full script is: do shell script "curl http://192.168.1.190/data_tmp/gespr_dat.csv.gz -s -o /Users/me/Desktop/gespr_dat.csv.gz"

Anybody any idea?

MBP, MM, MBP - 10.6 + Windooz XP on a hard partition

Posted on Oct 10, 2012 2:36 AM

80 replies

marshaul

Level 1

10 points

Oct 12, 2012 12:09 AM in response to ChangeAgent

Does nothing? It immediately drops you back to the command prompt?

< is a shortcut for the 'less' command. It's possible (probable?) this doesn't exist in your enviroment. Sorry about that.

Try

less /Users/me/Desktop/gespr_dat.csv.gz

sudo less /Users/me/Desktop/gespr_dat.csv.gz

Don't worry about this part too much, though. As I said, I'm fairly certain that it's an HTML file generated to inform you of lack of login credentials. I was just hoping to see the contents of it before we go any farther.

Check out that video in my last post. If you're having trouble understanding the html forms, perhaps it's time to start sharing files so I can take a look at things myself (specifically, the broken file, and all the HTML files served by the device).

ChangeAgent Author

Level 2

301 points

Oct 12, 2012 12:14 AM in response to marshaul

marshaul wrote:

Try using less (as I suggested in my post immediately prior to this one), and see what the contents of the file are.

Thnaks for the fast reply.

not sure what you mean here, I got the point about mailing it later but how do I use less when the download is already corrupted?

However, this is only to confirm that the nonworking file is, in fact, an HTML file.

At this point, however, I'm fairly certain that it is, and it's because it wants login credentials. Unfortunately, thanks to their using an HTML form,

I am convinced it is not a HTML file as (when not corrupted) I unzip it, it becomes a comma-separated values (CSV) file.

this will be a little more difficult than merely passing login credentials directly via curl.

OK I thought about this too, and since I am not good at code I went the easy way. I logged in via Safari and used curl to download the file. This route still left me with a unworkable file.

marshaul

Level 1

10 points

Oct 12, 2012 12:24 AM in response to ChangeAgent

OK, I'm confusing you by not being specific enough.

What I mean is, I'm fairly certain that the nonworking file is an HTML file.

less is a command line pager tool. A pager is basically a text viewer designed for minimum overhead and unconcerned with with file type (it interprets the content of the file as text and dumps it to the console.)

I want you to run less on the nonworking file, because that output dump will be the contents of the file, whatever they may be (although it will be unreadable if it isn't text.) This will allow me to confirm that it is, in fact, an HTML file.

OK I thought about this too, and since I am not good at code I went the easy way. I logged in via Safari and used curl to download the file. This route still left me with a unworkable file.

The problem is cookies. Cookies are where login information used by the server is stored, and are browser specific (curl cannot access Safari's cookies, just like if you switch to Chrome or Firefox or whatever you'll have to login to all your sites again.)

However, because curl is so awesome, they actually implemented cookies in it (the cookie jar feature). That is to say, curl can get and use its own cookies. This is precisely so that one can download files hidden by a login form like the one you're facing.

Check out that video.

ChangeAgent Author

Level 2

301 points

Oct 12, 2012 12:43 AM in response to marshaul

marshaul wrote:

less /Users/me/Desktop/gespr_dat.csv.gz

or

sudo less /Users/me/Desktop/gespr_dat.csv.gz

both work and give me:

<html><head><title>Document Error: Forbidden</title></head>

<body><h2>Access Error: Forbidden</h2>

<p>Bad state</p></body></html>

/Users/me/Desktop/gespr_dat.csv.gz (END)

Did you see my post about having the same result after I log in and sue curl?

Check out that video in my last post. If you're having trouble understanding the html forms, perhaps it's time to start sharing files so I can take a look at things myself (specifically, the broken file, and all the HTML files served by the device).

I did, I think I can follow it.

Message was edited by: ChangeAgent

marshaul

Level 1

10 points

Oct 12, 2012 12:59 AM in response to ChangeAgent

ChangeAgent wrote:

marshaul wrote:

less /Users/me/Desktop/gespr_dat.csv.gz

or

sudo less /Users/me/Desktop/gespr_dat.csv.gz

both work and give me:

<html><head><title>Document Error: Forbidden</title></head>

<body><h2>Access Error: Forbidden</h2>

<p>Bad state</p></body></html>

/Users/me/Desktop/gespr_dat.csv.gz (END)

See, it is an HTML file. =)

That "Access Error: Forbidden" actually was the contents of the file, interpreted as HTML. This is exactly what I wanted to know. Thanks.

Did you see my post about having the same result after I log in and sue curl?

Yup. I assume you have since seen my post where I explain this.

Incidentally, sudo is the unix command to run the command immediately following it with full admin (root) privileges. It stands for "super user do" (well, according to most people). It's used when a command returns something like the following, and you're really sure you want to do it anyway:

mv: rename foo to /usr/bin/foo: Permission denied

(That's me trying to move a dummy file "foo" into the /usr/bin directory without sufficient privileges.)

So, no need to use it if the command works the first time. And, be careful. Root is ultimate power. =)

ChangeAgent Author

Level 2

301 points

Oct 12, 2012 5:10 AM in response to marshaul

marshaul wrote:

Take a look at the following video:

http://www.linuxjournal.com/video/use-curl-login-websites-script

I did, and I do not get it to work. I am not sure if it is that the window that pops up seems to be a Java script or of something else.

I can also not find some of the bits that he suggests one need to write the curl script. My inspector (Safari) looks very different as to what he shows in the instructions.

her is the text, here I do not seem to find what to extract.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<head>

<title>Auerswald</title>

<!--[if lte IE 7]>

<![endif]-->

</head>

<body>

</div>

</div>

</div>

<h6>Zur Verwendung der Konfigurationsoberfläche muss Javascript in Ihrem Browser aktiviert sein.</h6>

</noscript>

</div>

</form>

</div> </div> </div> </div> </div> <div id="footer" style="z-index: 1;"> <div id="footer_content"> <div class="subcolumns"> <div id="infobar" class="c50l"> </div> <div id="toolbar" class="c50r"> </div> </div> </div> </div> <div id="overlay" style="display:none"></div> <div id="lightbox" class="lbDiv" style="display:none"></div> </body> </html>

FYI I am looking here:

ChangeAgent Author

Level 2

301 points

Oct 13, 2012 6:03 AM in response to ChangeAgent

OK I solved it. I found a web site with a Auerswald (maker of my switch) forum. My phone switch is a Auerswald 3000, firmware 4 FYI.

As I suspected the Java is part of the problem.

What I now have is (and it works):

outputfile=/Users/x/x/csv-$(date +%Y-Week%V).gz

cookie=/tmp/cookie.txt

auerswald=X.X.X.X

passwort=xxxxxx

curl -s --cookie-jar $cookie -o /dev/null -d "LOGIN_NAME=admin" -d "LOGIN_NOW=" -d "LOGIN_PASS=$passwort" http://$auerswald/login_json

curl -s --cookie $cookie -o /dev/null http://$auerswald/page_listgespr_export

curl -s --cookie $cookie --location -o $outputfile http://$auerswald/data_tmp/gespr_dat.csv.gz

curl -s --cookie $cookie -o /dev/null http://$auerswald/appclose

rm $cookie

Note, if you get here because you too have this problem read next bit to adjust for yourself.

outputfile=change to your own preferred location. Output file now has in my script the year and the week number, can be changed.

cookie=no need to change, but yo can

auerswald=your http here

passwort=you password

if you like me and was 'forced' to learn German in school you could read here:

Christian has the original idea and done a lot of the prep work, this script works but puts the file in your home folder and a .txt file to store the cookies.

http://cweiske.de/tagebuch/auerswald-gesprächsdatenliste.htm

here is a improvement on it (so I think).

http://www.jacotec.de/forum/archive/index.php/t-3963.html

the final solution I use is from zoo (12.08.2012, 08:54) to read it go way down.

Thanks for your help everybody.

marshaul

Level 1

10 points

Oct 13, 2012 7:58 AM in response to ChangeAgent

Glad you figured it out! And I didn't even have a chance to take a look at things since your previous post. =)

I love me a good shell script. 😎

ChangeAgent Author

Level 2

301 points

Oct 13, 2012 9:01 AM in response to marshaul

I am still implementing your suggestions and downloaded sendEmail. It does send out the mail (I am testing to myself) but the mail comes back (in to my email program).

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its

recipients. This is a permanent error. The following address(es) failed:

me@myisp.com

Unrouteable address

any idea what is going on here? Needles to say my email addres is correct.

marshaul

Level 1

10 points

Oct 13, 2012 9:19 AM in response to ChangeAgent

Can you post the entire command you're using?

ChangeAgent Author

Level 2

301 points

Oct 13, 2012 2:15 PM in response to marshaul

marshaul wrote:

Can you post the entire command you're using?

I send it via a POP account with my own ISP. It is a SSL connection.

SendEmail reports:

Oct 13 23:13:50 mbp sendEmail[61722]: Email was sent successfully!

My script is:

sendEmail -f me@myisp.com \

-t me@myisp.com \

-s smtprelaypool.ispgateway.com:465 \

-xu XXX \

-xp XYZ \

-u "Test email" \

-m "This is a test email."

marshaul

Level 1

10 points

Oct 13, 2012 9:55 PM in response to ChangeAgent

Well, the error message suggests some kind of DNS failure, but that's quite possibly misleading.

Are you sure about the outgoing server info (SMTP over port 465, as opposed to say 25 or 587)?

Perhaps you might try sending an email to another address (you have a google account or the like you can use as a test recipient?).

Are you quite sure that this email (i.e. from the same sender and to the same recipient, using the same server) can be successfully sent in other email clients?

Barring that, I don't see anything obviously amiss from what you've shown me. There are certainly other scriptable ways to do this. Try using the builtin mail command with uuencode.

Setup mail for use with smtp:

http://hints.macworld.com/article.php?story=20081217161612647

Various other ways to do this (mail + uuencode are listed):

http://www.panix.com/~kylet/unix-att.html

ChangeAgent Author

Level 2

301 points

Oct 14, 2012 3:29 AM in response to marshaul

OK worked it out. My wrong on the port number.

I can not send e-mail to myself using the application, even though I can from any other e-mail program.

I love your idea of using only terminal and am not not able to get terminal (OSX.6.8) to remove the file I download form my telephone switch to the trash after emailing it.

I tried the rm command followed with the path to the file, but get 'no such file sexists'. Do you know what command to use?

Thanks

marshaul

Level 1

10 points

Oct 14, 2012 3:59 AM in response to ChangeAgent

ChangeAgent wrote:

OK worked it out. My wrong on the port number.

I can not send e-mail to myself using the application, even though I can from any other e-mail program.

So, does that mean you got it to do what you needed it to? Or not? 😕

I tried the rm command followed with the path to the file, but get 'no such file sexists'. Do you know what command to use?

Well, rm is the command you want to use.

(Are you decompressing the .gz file, and then rm'ing it? If so, does rm work there? Perhaps you can compare the working rm command with non-working.)

If you're getting "no such file exists", you have to have typed the path wrong. (Possibly there are special characters like spaces in the path, somehow?) Or the file really doesn't exist. (Does it?)

One way you can be sure to get the right path is to locate the file you wish to delete in the Finder, and simply drag-and-drop it into the Terminal window. This will automatically place the correct absolute path in the prompt, complete with any escapes needed for special characters (if any).

So, if you want to test rm to satisy yourself, you can just go to the prompt and type "rm " (note the space) and then drag the file into the terminal window, and press return. Assuming this works, you can then copy and paste the path from the terminal window into your script.

marshaul

Level 1

10 points

Oct 14, 2012 11:47 AM in response to marshaul

Re: my last post: You can also use autocompletion. As you're typing a path, you can type the first few characters of the directory or file, and press tab. If you've typed enough that there is only one possible match, you'll get completion. If not, or if you mistyped something, you'll get a beep or a list of choices, depending on how your environment is set up.

(Frankly, I don't remember so much any more how it all comes out of the box. My shell environment – zsh – is anything but out of the box. In this case, it has advanced autocompletion which allows for errors and provides a selectable list if you haven't typed enough.)

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

do shell script "curl etc" not working properly.