cron for monitoring website changes

Question

Level 1

4 points

cron for monitoring website changes

I'm trying to use cron to monitor a website for changes and then send an email to myself when it detects a change but I can't seem to get it to work. Here's what my user crontab looks like:

*/10    *       *       *       *       cd /Users/USERNAME/Desktop/MonitorSite/ && curl -s http://www.example.com | md5 | tee new - | diff -q - old || echo 'A change was detected on Example.com' | mail -s 'Example.com has changed' my.email@gmail.com && mv new old

Right now I can't even get cron to run a simpler command that just emails a test message to myself every minute:

*/1    *       *       *       *       echo 'This is a test.' | mail -s 'Test' my.email@gmail.com

The command runs fine in the terminal if I execute it directly so I know that msmtp is setup properly and accessing my gmail password from Keychain, I just can't figure out why cron will not run it at the specified intervals. I would really appreciate any help and thanks in advance!

iMac, Mac OS X (10.7.2), 2.93 GHz, 16 GB RAM

Posted on Jan 2, 2012 2:29 PM

Reply

Answer 1

MrHoffman

Level 10

146,497 points

Jan 2, 2012 3:21 PM in response to icazzi

The cron tool has specific requirements for how you access the cron database; it's a little arcane, both the command syntax and the use of tabs.

I sometimes use periodic for this (and more details here), and a shell script rather than placing the stuff all into the cron table, as I've found an approach based on periodic and scripts easier than dealing (directly) with cron.

Apple prefers launchd for this and for newer applications; there are some good tools for managing launchd for these scheduled tasks (Lingon among those), or you can create the launchd database entry yourself.

Reply

Answer 2

Linc Davis

Level 10

209,739 points

Jan 2, 2012 3:39 PM in response to icazzi

Cron is not a shell and it can't process compound commands. You'd have to wrap the whole thing in a shell invocation (sh -c '...').

Reply

Answer 3

BobHarris

Level 9

56,939 points

Jan 2, 2012 4:51 PM in response to icazzi

The way you composed your command is OK, in that you used pipes, || and && to connect commands. I've done similar things myself. Although as Linc Davis suggests, it might be better to put this into a script that you invoke from cron, if only to make it cleaner and easier to modify.

However, your cron environment is not going to be anything like your terminal environment. My cron environment has a very minimum set of environment variables. It is possible that you are depending on something that doesn't exist in the cron environment.

I would suggest running a cron script that does the following:

/bin/pwd           >/tmp/cron.env.txt
/usr/bin/id -a    >>/tmp/cron.env.txt
echo $#           >>/tmp/cron.env.txt
/usr/bin/printenv >>/tmp/cron.env.txt

Reply

Answer 4

Camelot

Level 9

58,575 points

Jan 2, 2012 10:41 PM in response to icazzi

Don't try to run compound/piped statements in a crontab entry.

Instead, save your steps in a separate shell script file and just invoke that via cron.

For example, save those commands as a shell script /usr/local/bin/checksite.sh

#! /bin/sh

cd /Users/USERNAME/Desktop/MonitorSite/

curl -s http://www.example.com | md5 | tee new - | diff -q - old || echo 'A change was detected on Example.com' | mail -s 'Example.com has changed' my.email@gmail.com && mv new old

Then invoke this by adding /usr/local/bin/checksite.sh to the crontab.

Reply

Answer 5

icazzi Author

Level 1

4 points

Jan 3, 2012 9:37 AM in response to icazzi

Thanks to everyone for your help.

@MrHoffman

I will definitely be looking into using launchd in the future.

@Linc Davis, BobHarris, Camelot

As suggested, I have put the commands into a script that I invoke from cron. Mail is now being sent locally from cron detailing further issues which I suspect are related to BobHarris' point about the cron environment being different. Here is an example of the email:

USERNAME@imac:~$ mail

Mail version 8.1 6/6/93.  Type ? for help.

"/var/mail/USERNAME": 2 messages 2 new

>N  1 USERNAME@imac.local     Tue Jan  3 10:53  22/859   "Cron <USERNAME@imac> /Users/USERNAME/Desktop/"

N  2 USERNAME@imac.local     Tue Jan  3 10:54  19/703   "Cron <USERNAME@imac> /Users/USERNAME/Desktop/"

? t 1

Message 1:

From USERNAME@imac.local  Tue Jan  3 10:53:08 2012

X-Original-To: USERNAME

Delivered-To: USERNAME@imac.local

From: USERNAME@imac.local (Cron Daemon)

To: USERNAME@imac.local

Subject: Cron <USERNAME@imac> /Users/USERNAME/Desktop/MonitorSite/checksite.sh

X-Cron-Env: <SHELL=/bin/sh>

X-Cron-Env: <PATH=/usr/bin:/bin>

X-Cron-Env: <LOGNAME=USERNAME>

X-Cron-Env: <USER=USERNAME>

X-Cron-Env: <HOME=/Users/USERNAME>

Date: Tue,  3 Jan 2012 10:53:00 -0600 (CST)

/Users/USERNAME/Desktop/MonitorSite/checksite.sh: line 4: md5: command not found

Files - and old differ

send-mail: authentication method PLAIN needs a password

send-mail: could not send mail (account default from /usr/local/etc/msmtprc)

Here is what my script looks like:

#! /bin/sh

cd /Users/USERNAME/Desktop/MonitorSite/

curl -s http://www.example.com | md5 | tee new - | diff -q - old || echo 'A change was detected on Example.com' | mail -s 'Example.com has changed' my.email@gmail.com && mv new old

Lastly, here is the output from running the cron script that BobHarris suggested:

USERNAME@imac:~$ cat /tmp/cron.env.txt

/Users/USERNAME

uid=501(USERNAME) gid=20(staff) groups=20(staff),401(com.apple.access_screensharing),12(everyone),33(_appstore),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),100(_lpoperator),204(_developer),403(com.apple.sharepoint.group.2),402(com.apple.sharepoint.group.1)

SHELL=/bin/sh

USER=USERNAME

PATH=/usr/bin:/bin

PWD=/Users/USERNAME

HOME=/Users/USERNAME

SHLVL=2

LOGNAME=USERNAME

_=/usr/bin/printenv

Thanks again for the help.

Reply

Answer 6

icazzi Author

Level 1

4 points

Jan 3, 2012 9:40 AM in response to icazzi

Let me know if there is a better way to be posting code here.

Reply

Answer 7

Camelot

Level 9

58,575 points

Jan 3, 2012 9:44 AM in response to icazzi

Well, the most obvious problem:

/Users/USERNAME/Desktop/MonitorSite/checksite.sh: line 4: md5: command not found

As you can see from the other data in the email, the environment's $PATH is /usr/bin:/bin, so those are the only ywo directories the shell will search when looking for an executable, but md5 is in /sbin.

Therefore, at the very least, you'll need to edit your script to call /sbin/md5 rather than just 'md5'.

The mail authentication error seems like a red flag, too, but I haven't had enough coffee yet.

Reply

Answer 8

BobHarris

Level 9

56,939 points

Jan 3, 2012 10:20 AM in response to icazzi

icazzi wrote:

Let me know if there is a better way to be posting code here.

Click on the "Use Advanced Editor" (upper right corner of the edit box)

Click on the >> and select Syntax Highlighting -> Plain

It is not as good as the old [code] tags, but it is a start.

Reply

Answer 9

icazzi Author

Level 1

4 points

Jan 3, 2012 10:27 AM in response to icazzi

That cleared up the issue with md5 not being found but the mail authentication error is still there. If I call the script directly from the terminal it executes without issue and sends an email out to my gmail address when a change is detected.

In case this helps, here is my version of msmtp:

USERNAME@imac:~$ msmtp --version

msmtp version 1.4.26
Platform: x86_64-apple-darwin11.2.0
TLS/SSL library: OpenSSL
Authentication library: built-in
Supported authentication methods:
plain cram-md5 external login 
IDN support: disabled
NLS: disabled
Keyring support: MacOS 
System configuration file name: /usr/local/etc/msmtprc
User configuration file name: /Users/USERNAME/.msmtprc


Copyright (C) 2011 Martin Lambers and others.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

and here is what the configuration file looks like:

USERNAME@imac:~$ cat /usr/local/etc/msmtprc 

defaults
tls on
logfile ~/.msmtp.log


account my.email@gmail.com
host smtp.gmail.com
port 587
protocol smtp
auth on
from my.email@gmail.com
user my.email@gmail.com
tls on
tls_starttls on
tls_trust_file ~/Dropbox/Developer/Thawte Root Certificates/thawte Premium Server CA/Thawte Premium Server CA.pem

account default : my.email@gmail.com

I also have sendmail set to use msmtp:

USERNAME@imac:~$ cat ~/.mailrc 
set sendmail="/usr/local/bin/msmtp"

Reply

Answer 10

icazzi Author

Level 1

4 points

Jan 3, 2012 10:26 AM in response to BobHarris

Thanks, that is better.

Reply