Trouble with rsync and launchd

Question

Level 1

0 points

Trouble with rsync and launchd

I'm new to rsync and launchd. That being said, I have a computer at work that I would like to back up. I only need to back up a couple of folders, and I won't be connected to any external drives except for an 8GB usb stick. I don't want to use chronosync, time machine, or any other third party software. I'm not that kind of guy.

I have had reasonable success with this using rsync and launchd. But there are a few quirks. For instance, the following launch agent spawns TWO rsync processes that run continuously and eat up around 30% of my CPU. Does anybody know why there are two processes, why they run continuously, or why they eat up so much power when they SHOULD be copying nothing?

It seems to be copying all the files over and over again every ten minutes, despite the fact that they haven't changed. My entire reason for using rsync was to get avoid this.

----plist---
<plist version="1.0">
<dict>
<key>Label</key>
<string>USB backup</string>
<key>ProgramArguments</key>
<array>
<string>/usr/bin/rsync</string>
<string>-a</string>
<string>/Users/alan/Documents/school/</string>
<string>/Volumes/ALAN8G/backUp/school</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>StartInterval</key>
<integer>600</integer>
</dict>
</plist>
</code>
----/plist----

On a related note, I tried doing something very similar to this to backup my firefox profile data. No success. I think the space in /Application Support/ is messing with launchd. Thoughts?

Thanks in advance.

Macbook Pro 13", Mac OS X (10.6.2)

Posted on Dec 13, 2009 12:43 PM

Reply

Answer 1

twtwtw

Level 5

4,948 points

Dec 13, 2009 1:20 PM in response to ajeprog

well, first, you might think about updating rsync. the version of rsync that ships with os X (leopard and snow leopard) has some peculiarities when dealing with files with resource forks. it shouldn't really make a difference unless you have the extended attributes flag on (which you don't), but still...

rsync always seems to spawn multiple processes when it runs. I think it uses separate processes so that it can do different tasks (checking for differences, transferring files, etc.) asynchronously. It will also require some processor time to do checking tasks, even when it isn't transferring any files. do you have some reason to think that it's updating files rather than merely looking at them? checking shouldn't take long (on my backup script it takes it about 20 seconds to check through about 60 Gb of stuff). just as a thought, look in the /Volumes folder to make sure that you've got the path to the backup drive correct. sometimes (say if the drive gets disconnected) rsync will create a folder in /Volumes and write to that, which can slow things down and muck up your backups.

the run at load key is unnecessary, incidentally...

re firefox: paths in ProgramArguments keys don't need to be quoted or escaped; paths in Program keys do. if you're using a program key string, use single quotes or backslashes.

Message was edited by: twtwtw

Reply

Answer 2

Tony T1

Level 6

10,247 points

Dec 13, 2009 1:40 PM in response to ajeprog

On a related note, I tried doing something very similar to this to backup my firefox profile data. No success. I think the space in /Application Support/ is messing with launchd. Thoughts?

What's the error message? (look in Console)

Reply

Answer 3

ajeprog Author

Level 1

0 points

Dec 13, 2009 8:07 PM in response to Tony T1

Okay, I'm have more information.

I'm trying to run this command every 10 minutes:

/usr/bin/rsync -av /Users/alan/Documents/school/ /Volumes/ALAN8G/backUp/school/

So I loaded this plist into launchd:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" " http://www.apple.com/DTDs/PropertyList-1.0.dtd ">
<plist version="1.0">
<dict>
<key>Label</key>
<string>USB Backup School</string>
<key>ProgramArguments</key>
<array>
<string>/usr/bin/rsync</string>
<string>-av</string>
<string>/Users/alan/Documents/school</string>
<string>/Volumes/ALAN8G/backUp/school</string>
<string>>></string>
<string>/Users/alan/Documents/backUpLogs/schoolBackUp.log</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>StartInterval</key>
<integer>600</integer>
</dict>
</plist>

And I get these console messages:

12/13/09 10:02:24 PM USB Backup School[494] building file list ...
12/13/09 10:02:24 PM USB Backup School[494] rsync: link_stat "/Volumes/ALAN8G/backUp/school" failed: No such file or directory (2)
12/13/09 10:02:24 PM USB Backup School[494] rsync: link_stat "/>>" failed: No such file or directory (2)
12/13/09 10:02:24 PM USB Backup School[494] done
12/13/09 10:02:24 PM USB Backup School[494] ERROR: destination must be a directory when copying more than 1 file
12/13/09 10:02:24 PM USB Backup School[494] rsync error: errors selecting input/output files, dirs (code 3) at /SourceCache/rsync/rsync-37.3/rsync/main.c(503) [receiver=2.6.9]
12/13/09 10:02:24 PM USB Backup School[494] rsync: connection unexpectedly closed (8 bytes received so far) [sender]
12/13/09 10:02:24 PM USB Backup School[494] rsync error: error in rsync protocol data stream (code 12) at /SourceCache/rsync/rsync-37.3/rsync/io.c(452) [sender=2.6.9]
12/13/09 10:02:24 PM com.apple.launchd.peruser.501[84] (USB Backup School[494]) Exited with exit code: 12

Both of the relevant directories do exist, and the command works just fine when I run it in terminal. So what nuance of launchd am I missing?

Reply

Answer 4

twtwtw

Level 5

4,948 points

Dec 13, 2009 8:49 PM in response to ajeprog

three observations:

# your paths are different. in the command-line version you use trailing slashes ({color:green}/Users/alan/Documents/school/{color}), whereas in the plist you don't ({color:green}/Users/alan/Documents/school{color}). trailing slashes are significant for rsync.
# I doubt that redirecting standard out using >> works in ProgramArguments. it might work in a Program key, but I don't think it works here. instead, use the StandardOutPath and StandardErrorPath keys to specify the output file path.
# remove the RunAtLoad key, unless you intend for the script to run at system startup (when your USB drive probably won't be connected).

Message was edited by: twtwtw

Reply

Answer 5

Tony T1

Level 6

10,247 points

Dec 14, 2009 3:02 PM in response to twtwtw

2. I doubt that redirecting standard out using >> works in ProgramArguments. it might work in a Program key, but I don't think it works here. instead, use the StandardOutPath and StandardErrorPath keys to specify the output file path.

Why not just set an rsync log?
Put this as an rsync argument in the plist: --log-file=/Users/alan/rsync.log

Reply

Answer 6

Tony T1

Level 6

10,247 points

Dec 14, 2009 3:04 PM in response to ajeprog

Are you using a GUI to write the plist? If not, take a look at http://sourceforge.net/projects/lingon/files/

Reply

Answer 7

ajeprog Author

Level 1

0 points

Dec 14, 2009 3:10 PM in response to Tony T1

Tony T1 wrote:
Are you using a GUI to write the plist? If not, take a look at http://sourceforge.net/projects/lingon/files/

That list was generated by lingon.

Reply

Answer 8

Tony T1

Level 6

10,247 points

Dec 14, 2009 3:48 PM in response to ajeprog

If you replace
<string>>></string>
<string>/Users/alan/Documents/backUpLogs/schoolBackUp.log</string>
with
<string>--log-file=/Users/alan/Documents/backUpLogs/schoolBackUp.log<string>

What happens? (and what is reported in schoolBackUp.log?)

(Also, no need for -v ("-av") just use -a)

Reply

Answer 9

ajeprog Author

Level 1

0 points

Dec 15, 2009 12:30 PM in response to Tony T1

Thanks guys. The first part works now. Here is the final plist, for future reference:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" " http://www.apple.com/DTDs/PropertyList-1.0.dtd ">
<plist version="1.0">
<dict>
<key>Label</key>
<string>USB Backup School</string>
<key>ProgramArguments</key>
<array>
<string>/usr/bin/rsync</string>
<string>-av</string>
<string>/Users/alan/Documents/school/</string>
<string>/Volumes/ALAN8G/backUp/school/</string>
<string>--log-file=/Users/alan/Documents/backUpLogs/schoolBackUp.log</string>
</array>
<key>StartInterval</key>
<integer>600</integer>
</dict>
</plist>

Now on to my second problem, doing the same thing when the source path has a space in one of the directory names. Unfortunately, renaming the directory is right out because it's in Application Support. Here is what I have so far

Trying to run the command:

/usr/bin/rsync -a /Users/alan/Library/Application\ Support/Firefox/Profiles/9va0cbyb.default/ /Volumes/ALAN8G/backUp/9va0cbyb.default/ --log-file=/Users/alan/Documents/backUpLogs/firefoxBackUp.log

That works just fine in a terminal, so I'm having another formatting issue in my plist. Here's what I have for that:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" " http://www.apple.com/DTDs/PropertyList-1.0.dtd ">
<plist version="1.0">
<dict>
<key>Label</key>
<string>USB Backup Firefox</string>
<key>ProgramArguments</key>
<array>
<string>/usr/bin/rsync</string>
<string>-av</string>
<string>'/Users/alan/Library/Application Support/Firefox/Profiles/9va0cbyb.default/'</string>
<string>/Volumes/ALAN8G/backUp/9va0cbyb.default/</string>
<string>--log-file=/Users/alan/Documents/backUpLogs/firefoxBackUp.log</string>
</array>
<key>StartInterval</key>
<integer>600</integer>
</dict>
</plist>

Which gives me these console messages:

12/15/09 2:20:52 PM USB Backup Firefox[666] building file list ... rsync: link_stat "/'/Users/alan/Library/Application Support/Firefox/Profiles/9va0cbyb.default/'" failed: No such file or directory (2)
12/15/09 2:20:52 PM USB Backup Firefox[666] done
12/15/09 2:20:52 PM USB Backup Firefox[666] sent 29 bytes received 20 bytes 98.00 bytes/sec
12/15/09 2:20:52 PM USB Backup Firefox[666] total size is 0 speedup is 0.00
12/15/09 2:20:52 PM USB Backup Firefox[666] rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-37.3/rsync/main.c(992) [sender=2.6.9]
12/15/09 2:20:52 PM com.apple.launchd.peruser.501[83] (USB Backup Firefox[666]) Exited with exit code: 23

I'm reasonably certain the problem is on this line

<string>'/Users/alan/Library/Application Support/Firefox/Profiles/9va0cbyb.default/'</string>

I've tried using single quotes and escaping the space with a backslash. Neither worked and both gave similar console messages, ie "no such file or directory". So what's the problem?

Again, thanks for the help.

Reply

Answer 10

twtwtw

Level 5

4,948 points

Dec 15, 2009 12:47 PM in response to ajeprog

when using ProgramArguments, each string segment is quoted by launchd as needed. I suspect what you're ending up with is {color:green}''/Users/alan/Library/Application Support/Firefox/Profiles/9va0cbyb.default/''{color}, which will get treated as though there were no single-quotes at all. remove the single-quotes from that line and see if it fixes the problem.

Reply

Answer 11

ajeprog Author

Level 1

0 points

Dec 15, 2009 1:13 PM in response to twtwtw

twtwtw wrote:
when using ProgramArguments, each string segment is quoted by launchd as needed. I suspect what you're ending up with is {color:green}''/Users/alan/Library/Application Support/Firefox/Profiles/9va0cbyb.default/''{color}, which will get treated as though there were no single-quotes at all. remove the single-quotes from that line and see if it fixes the problem.

Works like a charm! Thanks for all the help.

Reply