Python Scripts in Virtual Environments in Background macOS

rothnd wrote:

So I have a group of scripts run by the main.py script of daps. Right now my problem is with using a plist to load main.py so it's always running/accessible and so that it has access to an rclone exec.

What is all of that? What is daps? What is rclone?

What do you mean by "accessible"? You can run some script, that's fine. But is said script providing some kind of service? What is "accessing" it? And how?

Is rclone this thing? https://github.com/rclone/rclone "rsync for cloud storage" - because rsync is just a pleasure to use elsewhere. </sarcasm>

I invoke the main.py manually with

cd /Users/nathan/daps

source .venv/bin/activate

python3 /Users/nathan/daps/main.py

OK. Now we're getting somewhere.

which seems to work fine. However, when I attempt to load main.py in the background and it makes its call to rclone, rclone fails (command not found).

So this daps thingy is supposed to call some other open-source thingy to do something with some 3rd party cloud service? If you ever get this running, maybe wrap it all up into an app and sell it as "Masochist Cloud - the media manager choice for people who want to punish themselves."

My updated plist

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

<dict>

<key>Label</key>

<string>com.nathan.daps</string>

<key>ProgramArguments</key>

<array>

<string>sh</string>

<string>-c</string>

<string>.venv/bin/python3 main.py</string>

</array>

<key>WorkingDirectory</key>

<string>/Applications/daps</string>

</dict>

</plist>

That's never going to work.

You have to go back your manual execution and figure out what that's doing.

cd /Users/nathan/daps

Change the working directory to the daps directory in your home directory. OK. Fair enough.

source .venv/bin/activate

This is more complicated. It's going to stuff a bunch of stuff into your environment. What kind of stuff?

python3 /Users/nathan/daps/main.py

Run python3 (presumably found in some path stuffed in via step 2) and execute this python script.

That's 3 steps. You can't do that in a launchd script. You could kinda do #1 via WorkingDirectory, but that's such a cop-out. But regardless, you can't do those other 2 steps. I don't know. Maybe you do something funky with "sh" and semicolons. How about we do it properly instead?

What you have to do is merge these into a single script. The script is pretty simple. First it changes to the desired directory. Then it runs "source .venv/bin/activate" and then it runs "python3 /Users/nathan/daps/main.py". Save the three-line script as something named "daps.sh" in some convenient place. Probably shouldn't put it in the existing "daps" directory though.

If you want to be really fancy, you could add a shebang line and make the script executable with "chmod". Then you could just run the script with nothing else. Otherwise, you'll still have to run "sh" to execute it. Since it's a stand-alone script, you don't need the "-c" parameter.

Your plist would look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>me.nathan.daps</string>
 <key>ProgramArguments</key>
 <array>
     <string>/bin/sh</string>
     <string>/Users/nathan/bin/daps.sh</string>
 </array>
</dict>
</plist>

rothnd wrote:

If you look at that first kometa plist I posted. That's what's not working for me at the moment.

Of course not. As I said, it was always wrong. It would never have worked.

Both of these scripts have 'schedulers' so in theory the script is always running (start, login, etc) so that when the bell tolls it does its thing. I've been sorely confused on scheduling since I was just starting to use cron like last year

I'm not sure what you mean by "scheduler" in this context. It's simply too general of a word. But it definitely sounds like you are talking about two completely different and unrelated types of "schedulers".

If these Python scripts need to be always running, then cron is not appropriate. They should be launched at system start time instead. System start time typically involves system facilities, the root user, or, even better, an unprivileged, dedicated user. But these are advanced topics to it's better to start with a task that runs when a user beings a login session. Once you have that part working a normal user context, then you could move it if you wanted to.

The cron architecture is designed to run scripts on your local machine according to a given schedule. Maybe you want to run something every day at midnight, the 4th Thursday of each month, or every 17 seconds.

then it recently stopped working

Why is that? That's a rhetorical question, of course. What I mean is that you've technically over-extended yourself. There is a fancy term for it called "technical debt". Just like financial debt, the only solution is to pay it off. You can't fix it with additional debt.

these people routinely do stuff like that for tutorials and I'm left trying to decipher.

Don't bother. The internet is fake. They're simply lying to you. This was something you could have done a few years ago. But these days, it isn't possible. There is some legitimate content available, but only those who already know these technical details can identify what is fake and what is legitimate. And most of those people could probably do the given task themselves. It's a tricky thing, navigating disinformation in the modern world.

Basically, these 'projects' get confined to their own directories. I point Terminal at that directory. Create a python virtual environment for isolation. And run the scripts. One problem I'm having is in these virtual environments. source activate...do stuff....deactivate. But when I'm 'activated' in a Terminal session the Terminal won't release until deactivate. I'm trying to reconcile this 'release/deactivate' with a script running on a schedule needing to be 'always available'.

The confusion and misinformation builds upon each other.

The first thing to understand is that Terminal really has nothing to do with scripting. Terminal is an interactive command line environment. Scripting is a non-interactive environment. Probably their only real similarity with each other is that they both facilitate command-line program execution with arguments, as opposed to double-clicking an object in the Finder. Other than that one specific aspect, the only commonality they share is social - a community of practice. People who use one, tend to also use the other.

There is never any need to "deactivate" anything. That's simply an artifact of the "source" command when run in Terminal. It's normal for a script or executable to require very specific environment settings like PATH, etc. Normally a script would setup what it needs, and then run the desired command. When the command completes, the entire modified environment goes away. There's no need for it anymore.

But I would like to learn.

That's good then. Just remember that not everyone is honest. And dishonesty thrives on the internet. There are no laws, so anything goes. Now, with modern AI systems, they can generate these fake tutorials in seconds. They look correct. In many cases they may be 90% correct. But you can only learn by doing yourself.

Ideally, a structured, human-led, interactive, learn-by-doing practice is best. The material is structured so that it is always just a little bit more difficult than you can handle. That way, you don't get lost and you don't get bored. Back in my day, we used to call these things schools, colleges, and universities.

...to be continued...

Sorry, 5000 character limit per post.

rothnd wrote:

Specifically, you were saying something about just an executable script for this. So in plain terms, create exec script to run these virtual environments, add to login items (I'm assuming. No need for launchd?). How then do I handle the whole Terminal won't release (or needs to always have the window open) until deactivate?

As before, the problem is social, not technical.

Traditionally, the Mac community of practice involved running apps by double-clicking them. Login Items are apps that will run automatically at login without having to be manually double-clicked.

The Unix community of practice involved writing scripts, setting up environments, and running those scrips as cron jobs and/or system services.

When Apple made Mac OS X 25 years ago, they threw these two communities into the same box and they've been fighting ever since. It is possible to create wrapper apps that run command-line tools. It's also possible to create purely Mac-style (now iOS-style) scripts using AppleScript, Automator, and now Shortcuts. But usually, that's extra work. You have to get the command-line version working first. And in this case, wrapping it into a Login Item-friendly app probably isn't going to do anything for you.

However, I don't really know what to tell you at this point. My post the other day explained everything you needed to do. Create a single script that sets up your environment and then executes your task. You can test that in Terminal. But it has to be a one-liner. You can wrap it via "/bin/sh -c" or you can use "chmod" and shebang to make your script behave like a stand along executable. Either way is fine. But that one-liner can then be run via launchd.

rothnd wrote:

Title says what I'm trying to accomplish.

It does not.

Using 2 services - 1. wiki Drazzilb08/daps and 2. local Kometa.

I went to both site and I still have no idea what these things do.

These install pretty similarly and are invoked similarly but I can't seem to blend them together.

Blend them together in what way? What do they do?

I have tried too many things to enumerate here but they include: running the python command with nohup, running with disown, making a LaunchAgent plist that calls a sh script to run the commands, all to no avail.

The first step is always getting something running properly in an interactive Terminal session. Any kind of background operation is going to be wildly different. But that doesn't change the fact that you have to be able to run it manually first. Until you can do that, you will never make any further progress.

Assuming you can run them manually, please describe how you would do that, and what that would do for you.

I don't understand what this one does so...

Tell me about it.

The plist I'm trying to understand

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

<dict>

<key>Label</key>

<string>com.YOUR_USERNAME.kometa</string>

<key>ProgramArguments</key>

<array>

<string>sh</string>

<string>-c</string>

<string>kometa-venv/bin/python kometa.py --config /path/to/kometa-config/config.yml</string>

</array>

<key>UserName</key>

<string>YOUR_USERNAME</string>

<key>WorkingDirectory</key>

<string>/PATH/TO/KOMETA</string>

</dict>

</plist>

Which gets invoked peculiarly to me

cd ~/Library/LaunchAgents/

launchctl bootstrap gui/YOUR-USER-ID com.YOUR_USERNAME.kometa.plist

And doesn't seem to work.

Of course not. It's wrong in at least 3 different ways.

How can I best accomplish 2 python scripts in background in their respective virtual environments at startup running macOS 14.7.2

At startup or login? Are these services used by a single user, or are they supposed to be available to all users?

What command lines would you execute if you just wanted to run it manually in Terminal?

Is that daps program interactive? I have no idea if that will work. I've never heard of a command-line tool that runs as a service and is also interactive. Normally a service runs, in the background, without any interaction. If you do need interaction, there will be one or more client tools, or maybe even a web interface, to make any interactive adjustments.

This is something that is out of scope for these Apple communities but is potentially better served at the Python community or Apple's Developer forums.