Solving problem is about exposing yourself to as many situations as possible like How to start a background process in Python? and practice these strategies over and over. With time, it becomes second nature and a natural way you approach any problems in general. Big or small, always start with a plan, use other strategies mentioned here till you are confident and ready to code the solution.
In this post, my aim is to share an overview the topic about How to start a background process in Python?, which can be followed any time. Take easy to follow this discuss.
I’m trying to port a shell script to the much more readable python version. The original shell script starts several processes (utilities, monitors, etc.) in the background with “&”. How can I achieve the same effect in python? I’d like these processes not to die when the python scripts complete. I am sure it’s related to the concept of a daemon somehow, but I couldn’t find how to do this easily.
Note: This answer is less current than it was when posted in 2009. Using the
subprocess module shown in other answers is now recommended in the docs
(Note that the subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using these functions.)
If you want your process to start in the background you can either use
system() and call it in the same way your shell script did, or you can
import os os.spawnl(os.P_DETACH, 'some_long_running_command')
(or, alternatively, you may try the less portable
See the documentation here.
While jkp‘s solution works, the newer way of doing things (and the way the documentation recommends) is to use the
subprocess module. For simple commands its equivalent, but it offers more options if you want to do something complicated.
Example for your case:
import subprocess subprocess.Popen(["rm","-r","some.file"])
This will run
rm -r some.file in the background. Note that calling
.communicate() on the object returned from
Popen will block until it completes, so don’t do that if you want it to run in the background:
import subprocess ls_output=subprocess.Popen(["sleep", "30"]) ls_output.communicate() # Will block for 30 seconds
See the documentation here.
Also, a point of clarification: “Background” as you use it here is purely a shell concept; technically, what you mean is that you want to spawn a process without blocking while you wait for it to complete. However, I’ve used “background” here to refer to shell-background-like behavior.
You probably want the answer to “How to call an external command in Python”.
The simplest approach is to use the
os.system function, e.g.:
import os os.system("some_command &")
Basically, whatever you pass to the
system function will be executed the same as if you’d passed it to the shell in a script.
I found this here:
On windows (win xp), the parent process will not finish until the
longtask.py has finished its work. It is not what you want in CGI-script. The problem is not specific to Python, in PHP community the problems are the same.
The solution is to pass
DETACHED_PROCESS Process Creation Flag to the underlying
CreateProcess function in win API. If you happen to have installed pywin32 you can import the flag from the win32process module, otherwise you should define it yourself:
DETACHED_PROCESS = 0x00000008 pid = subprocess.Popen([sys.executable, "longtask.py"], creationflags=DETACHED_PROCESS).pid
subprocess.Popen() with the
close_fds=True parameter, which will allow the spawned subprocess to be detached from the Python process itself and continue running even after Python exits.
import os, time, sys, subprocess if len(sys.argv) == 2: time.sleep(5) print 'track end' if sys.platform == 'darwin': subprocess.Popen(['say', 'hello']) else: print 'main begin' subprocess.Popen(['python', os.path.realpath(__file__), '0'], close_fds=True) print 'main end'
You probably want to start investigating the os module for forking different threads (by opening an interactive session and issuing help(os)). The relevant functions are fork and any of the exec ones. To give you an idea on how to start, put something like this in a function that performs the fork (the function needs to take a list or tuple ‘args’ as an argument that contains the program’s name and its parameters; you may also want to define stdin, out and err for the new thread):
try: pid = os.fork() except OSError, e: ## some debug output sys.exit(1) if pid == 0: ## eventually use os.putenv(..) to set environment variables ## os.execv strips of args for the arguments os.execv(args, args)
Both capture output and run on background with
As mentioned on this answer, if you capture the output with
stdout= and then try to
read(), then the process blocks.
However, there are cases where you need this. For example, I wanted to launch two processes that talk over a port between them, and save their stdout to a log file and stdout.
threading module allows us to do that.
First, have a look at how to do the output redirection part alone in this question: Python Popen: Write to stdout AND log file simultaneously
#!/usr/bin/env python3 import os import subprocess import sys import threading def output_reader(proc, file): while True: byte = proc.stdout.read(1) if byte: sys.stdout.buffer.write(byte) sys.stdout.flush() file.buffer.write(byte) else: break with subprocess.Popen(['./sleep.py', '0'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc1, subprocess.Popen(['./sleep.py', '10'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc2, open('log1.log', 'w') as file1, open('log2.log', 'w') as file2: t1 = threading.Thread(target=output_reader, args=(proc1, file1)) t2 = threading.Thread(target=output_reader, args=(proc2, file2)) t1.start() t2.start() t1.join() t2.join()
#!/usr/bin/env python3 import sys import time for i in range(4): print(i + int(sys.argv)) sys.stdout.flush() time.sleep(0.5)
stdout get updated every 0.5 seconds for every two lines to contain:
0 10 1 11 2 12 3 13
and each log file contains the respective log for a given process.
Tested on Ubuntu 18.04, Python 3.6.7.