Running a process automatically when the Mac restarts

Knut Harald Støre wrote:

I have tried using launchctl to make it start automatically, but it’s not working.

...

How can I make this work from crontab?

...

How can I make this work inside an Applescript?

First of all, you're going to have to pick one method and stick with it. All of these are going to be difficult for someone who's never done this before. All of these are radically different approaches.

I just followed someone else’s instructions online. I contacted him about this, but he couldn’t figure out why it didn’t work on my system.

Please never do that. As difficult as it might be to help someone figure something out here in the forums, we can't also try to debug some other random instructions from the internet, along with instructions on how to undo the damage caused by said random internet instructions. I don't care what you've heard, or what people on the internet have told you, the internet is wrong.

If I assume that anything you've tried thus far hasn't irreparably damaged your system, here is what you need to do.

Note that, at a bare minimum, the task you've been given by the developers of this zigbee2mqtt software is very difficult. It's built upon decades of software and system admin practices on Linux that aren't even good practices on Linux. Now you're trying to port all of that to a Mac. This is a difficult task even for people very experienced. They usually fail.

First of all, you want to use launchctl. This isn't necessarily the easiest solution. Your solution is already difficult. This is simply the solution that requires the least amount of additional expertise on top of what you already have to do.

Write a little script that sets up your environment and runs the script. When you run Terminal, your environment has already been setup. The launchctl environment is different. The easiest way to bridge that gap is to have launchctl directly run a shell environment in much the same way that Terminal does. Hopefully this will be enough.

So you want a zsh script that runs the following:

cd /Applications/zigbee2mqtt/

npm start

So create a file named zigbee2mqtt.sh with the contents of the above file. Put the file in a convenient, dedicated location such as: $HOME/Library/Scripts. You may have to create this directory. You could put it directly in your home directory too, but there is a chance it could get accidentally removed. The hidden Library folder is designed to be less likely to be damaged like this.

You'll also want to correct those commands to be more shell-script friendly. Change "npm" that you specify the full path to npm. I have no idea what that path should be on your computer. It all depends on how you installed it. This way, your script is independent of whatever path you have set at the time. This doesn't guarantee the script will survive accidents, but it's good practice.

Try to manually run your script by doing "sh /Users/<your short user name>/Library/Scripts/zigbee2mqtt.sh". If that works, then go to the next step. (Replace "<your short user name>" with what that is.)

Create a launchd plist file named something like "org.zigbee2mqtt.plist". Do not create the file inside Library/LaunchAgents. Create it somewhere else and move it into that folder only when it is ready to go.

The code should look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.zigbee2mqtt</string>
    <key>ProgramArguments</key>
    <array>
        <string>/bin/sh</string>
        <string>/Users/<your short user name>/Library/Scripts/zigbee2mqtt.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

again, replace "<your short user name>" with what that is.

Please note that this is extremely delicate and difficult to test. I don't want to log out of my system, so I tested it with:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.zigbee2mqtt</string>
    <key>ProgramArguments</key>
    <array>
        <string>/bin/zsh</string>
        <string>/Users/jdaniel/Library/Scripts/zigbee2mqtt.sh</string>
    </array>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>9</integer>
        <key>Minute</key>
        <integer>33</integer>
    </dict>
</dict>
</plist>

Where my zigbee2mqtt.sh consists of:

date >> /tmp/mydate.txt

Your script is significantly more difficult. I strongly recommend trying something simple like this until you get it working. Run:

launchctl bootstrap gui/502 $HOME/Library/LaunchAgents/zigbee2mqtt.plist

to load it and check /tmp/mydate.txt

and then run:

launchctl bootout gui/502/org.zigbee2mqtt

to unload it, make some fixes, and try again.

Knut Harald Støre wrote:

I see now that I have missed a few details like adding the –c option in your last example. I don’t know how important that is. I didn’t use it when running the script manually.

You don't need it when running the script manually. The -c option takes the next command line argument and executes it as if you had typed it in an interactive shell. It is necessary to have the "/bin/sh" command interpret the HOME environment variable as your home directory. You don't need the -c option when you run the script manually because you already have a shell environment in that case.

A side effect of this is that the script has to be executable. Technically my demo.sh script is wrong because I should have had a "shebang" line in there specifying which shell I want to use to run this script.

It is actually rare to have a single word after a -c option. Normally, there would be something more complicated in quotes. Although in a launchd plist file, you don't need to add the quotes. The XML delimiters do that for you.

Maybe there’s something fundamentally wrong with launchctl on my system.

There is not. It could be a homebrew problem. It might also be inexperience with launchctl. You do have to load the script via "launchctl bootstrap" and then unload it via "launchctl bootout" with each and every test. If you neglect to that, then the old version will still be loaded and it will refuse to reload it, issuing an unhelpful I/O error.

On Apple platforms, there is zero help from Apple regarding these kinds of low-level tools. The expectation is that if you can afford the top-of-the-line Apple products, then you can also afford top-of-the-line peripherals that work with them. If your floor heating system doesn't work with HomeKit, then you rip it out and install a new floor heating system that does. Or just buy/build a new house if you're impatient. HomeKit definitely shares the same socio-economic market strata as $1000 caster wheels and $1300 monitor stands. For developers like me, the attraction to the Apple platform isn't the great command-line tools or easy-to-use platform, it's the typical Apple product owner's disposable income.

Knut Harald Støre wrote:

I don’t use the HOME environment variable, instead specifying the complete path, but I have now added the –c option as well.

If you provide the full path, then you don't need the "-c" option. But shell scripts can be really tricky. I don't know what commands you are ultimately running. It's possible that you may need to make the script executable and include the "-c" option just to ensure that you have a full user environment available. Otherwise, it may run differently, or not at all. These kinds of things just take lots of trial and error.

Until now, I have not used the bootout command. Since the bootstrap command always failed in the first place, I figured there was no need. And now when I try the bootout command, it fails telling me there is no such process running.

It's not that there is no need. It's that you are expected to know all details of how the launchd architecture works. If you don't, then it's going to take you a lot more rounds of trial and error to figure it out.

Those bootstrap and bootout commands are for manually loading and testing. The boot process (or logging in for launch agents) performs the same operation as bootstrap. Shutting down, or restarting, (or logging out for launch agents) may be the same as a bootout. I have to qualify that with "may" because Apple doesn't always bother with clean shutdowns in many cases.

What this means is that you should use bootstrap and bootout to test. Only restart (or logout) when you're happy with the behaviour and you want to try it for real.

I went through trying to automatically launch my script a while back. Start with something extremely simple such as an a very simple basic AppleScript which you save as an app. I found that it was the most compatible & universal since other scripts/apps may add more complexity to it.

I also recall trying to load & launch the script using many variations of the "launchctl" command as I see you are doing. Read the "launchctl" man page very carefully since there may be some clues. Unfortunately there is very little information & details there. I also search online for more tidbits (most were too old to be useful these days, but they did provide a bit of insight).

You also need to make sure you have given your app security permissions to run. If something in that app uses some command line utilities, then those may need special permissions in the Privacy & Security System Settings. Unfortunately the Privacy & Security System Settings may place those items in there disabled without any notice. Plus those entries may be common names which can appear there for other apps. For my Python script I had to figure out which of multiple "python" entries in the Privacy & Security Settings was associated with my script.

The alternative is to use "cron" and "crontab" which is the traditional *nix method. I have seen a third part developer post on this forum that this is the best & easiest option.

You may want to check the system mail which may have more detailed information about the reason for the failure. The system mail is accessed by using the "mail" command (if you are not familiar with it, then you will need to read up on how to navigate its text interface since it is confusing).

Unfortunately I'm not at the computer where I was performing the experiments so I cannot provide any specific details on how I got it to work (or mostly work). In the end I was focusing on "crontab" since I thought that made the most sense for my case since I also needed to easily clean up after myself once the task was completed. I do recall during my tests that many times my automated Daemon/Agent/cronjob did launch, but halted since I had a component within a home user folder....I finally got that sorted as well (I think even for the Daemon/Agent).

I'll see if I can get more specific details once I'm on my other computer.

OK. I'm now sated. Since I'm no longer preoccupied with baser needs like hunger, I can think more clearly and try things out.

When running shell scripts on the command line interactively, it's not necessary for them to be executable. But with launchd, it is necessary. To do this, I run:

chmod u+x $HOME/Library/Scripts/demo.sh

Now, I can try a launchd plist file that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.demo</string>
    <key>ProgramArguments</key>
    <array>
        <string>/bin/sh</string>
        <string>-c</string>
        <string>$HOME/Library/Scripts/demo.sh</string>
    </array>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>10</integer>
        <key>Minute</key>
        <integer>26</integer>
    </dict>
    <key>StandardOutPath</key>
    <string>/tmp/out.txt</string>
    <key>StandardErrorPath</key>
    <string>/tmp/error.txt</string>
</dict>
</plist>

The "StandardOutPath" and "StandardErrorPath" are not necessary. They are just something to help with debugging. That's how I found out that the environment variable actually was working. It was a permissions problem.

Also, in my previous reply, I glossed over the "launchctl" commands. Apple "improved" them a few years back. So you now need to know your exact user ID and a target domain. In my case, my fallback admin user ID is "501". But my normal user ID is "502". I just happen to know that "gui" is the appropriate domain. Don't use the "user" domain. That's not (quite) right.

Also, I didn't actually answer the OP's question. The question was "Running a process automatically when the Mac restarts". All of this is to run a command when the user logs in. The difference between restarting and logging in isn't just a can-o-worms, it's a 55-gallon drum-o-worms. Can we all just assume that login on a consumer-grade macOS system is essentially the same as startup? Please?

It is possible to do all of this as a launch daemon, but then we can't use any of the environment variable or any user-specific paths. We also have to assume that all of that npm and "zigbee" stuff is accessible outside of a user environment. I didn't want to deal with all of that.

And last but not least, the elephant in the room. That which is never spoken about publicly, yet always lurks in the dark underbelly of any discussion of command-line things that don't work after soliciting instructions on the internet. If the poor OP has installed the notorious "Homebrew" software, then I disavow this thread in its entirely and commit the OP to their fate. All I could recommend in that case is a factory reset of the device, erasing the hard drive and reinstalling the operating system as penance.

@etresoft is the expert here and he has you covered. In fact I'm going to bookmark his post here so I can write myself some notes.

I suggest experimenting with something simple so that you know you have the basics working as expected, best to use a utility built into macOS. Then take that as a starting point to try to get what you actually want launched. As @etresoft mentioned, there are a ton of very small things that can easily derail things here. Having a full path for everything, access to environmental variables, and proper permissions are all going to play a part in your endeavor. I easily spent an entire day trying to get my script to launch as a Launch Agent and I only succeeded by starting with something even more basic than my Python script. Once I had something basic working, I was able to address the additional issues associated with my actual Python script (such as allowing the Python instance associated with my script to have security permissions).

At one point I noticed I had a message about having new system mail as I was using the command line (probably while trying crontab). If any system mail appears, then it will likely have some useful details that may help in troubleshooting things.

Keep in mind @etresoft has been a developer dealing with Apple for years. He definitely has a much better understanding about how the underlying system works than most anyone else on this forum. And even @etresoft doesn't find it easy to do this without trial & error. Most of the information you will find on the Internet is severely outdated so concentrate on the tips @etresoft has provided here since they are consolidated to one place....trust me that nothing else on the Internet will help you to use a Launch Daemon/Agent.

While it is a bit late now, it is always best to experiment on a test system first before you do so on the main production system. Having an extra Mac or a VM instance are best so that if you make a catastrophic mistake, you can just start completely over (VM is easiest to delete & use a new fresh instance). Besides, once you have got it working, I would still highly recommend testing on a clean system just to make sure you document the exact steps needed to make it work. You may find that you are missing a step from something you tried earlier, or you may find that you can strip out some steps. At some point you may need to recreate the setup or fix it once a macOS update/upgrade changes things.

Good luck.

Have you tired to add the app to the System/Login Items & Extensions/Open at Login settings pane so that it launches each time the Mac is booted?

Also note that I can't even describe how to do this in the 5000 character post limit here in the forums!

continued from before....

Note that I couldn't even get it to work with "$HOME" inside the plist file. I think this is possible, but I'm already late for breakfast. I'm also not testing "runatload" because I don't want to log out and back in each time to test. Once it's running, you should be able to just remove that "StartCalendarInterval" and replace with "RunAtLoad".

Ideally, you would name the file "org.zigbee2mqtt.plist" to match the label, but I forgot to do that.

If you can't get the above to work, then you just need to keep trying. I'm very experienced at this sort of thing and it took me 10 attempts before I gave up trying to get "$HOME" to work.

In the future, I strongly recommend using apps that simply work and don't require this level of command-line hacking. This is absolutely typical of this kind of open-source junk ware.

Knut Harald Støre wrote:

I do think I installed homebrew as part of that process.

If it's working, that's fine. But be warned that if you upgrade that computer to macOS 16 next year, you could have problems. Those major upgrades are normally what breaks Homebrew. And since people in that spot are normally following instructions from the internet, they have no idea how to fix it.

Thanks for the background on Zigbee. That's fascinating. I guess the few experiences I've had with HomeKit-style devices is pretty typical. Nothing works together. People scream bloody murder about Apple's $99 developer fee. But $5000 just to get started with Zigbee is much more reasonable I guess. 😄 I wouldn't touch that with a ten-foot pole. I guess other developers feel the same way. That's why you'll have to hack these things up on your own.