How can I print and display subprocess stdout and stderr output without distortion?

Posted on

Question :

How can I print and display subprocess stdout and stderr output without distortion?

Maybe there’s someone out in the ether that can help me with this one. (I have seen a number of similar questions to this on SO, but none deal with both standard out and standard error or deal with a situation quite like mine, hence this new question.)

I have a python function that opens a subprocess, waits for it to complete, then outputs the return code, as well as the contents of the standard out and standard error pipes. While the process is running, I’d like to also display the output of both pipes as they are populated. My first attempt has resulted in something like this:

process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

stdout = str()
stderr = str()
returnCode = None
while True:
    # collect return code and pipe info
    stdoutPiece =
    stdout = stdout + stdoutPiece
    stderrPiece =
    stderr = stderr + stderrPiece
    returnCode = process.poll()

    # check for the end of pipes and return code
    if stdoutPiece == '' and stderrPiece == '' and returnCode != None:
        return returnCode, stdout, stderr

    if stdoutPiece != '': print(stdoutPiece)
    if stderrPiece != '': print(stderrPiece)

There’s a couple problems with this though. Because read() reads until an EOF, the first line of the while loop will not return until the subprocess closes the pipe.

I could replace the read() in favor of read(int) but the printed output is distorted, cut off at the end of the read characters. I could readline() as a replacement, but the printed output is distorted with alternating lines of output and errors when there are many of both that occur at the same time.

Perhaps there’s a read-until-end-of-buffer() variant that I’m not aware of? Or maybe it can be implemented?

Maybe it’s best to implement a sys.stdout wrapper as suggested in this answer to another post? I would only want to use the wrapper in this function, however.

Any other ideas from the community?

I appreciate the help! 🙂

EDIT: The solution really should be cross-platform, but if you have ideas that aren’t, please share them away to keep the brainstorming going.

For another one of my python subprocess head scratchers, take a look at another of my questions on accounting for subprocess overhead in timing.

Asked By: perden


Answer #1:

Make the pipes non-blocking by using fcntl.fcntl, and use to wait for data to become available in either pipe. For example:

# Helper function to add the O_NONBLOCK flag to a file descriptor
def make_async(fd):
    fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)

# Helper function to read some data from a file descriptor, ignoring EAGAIN errors
def read_async(fd):
    except IOError, e:
        if e.errno != errno.EAGAIN:
            raise e
            return ''

process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

stdout = str()
stderr = str()
returnCode = None

while True:
    # Wait for data to become available[process.stdout, process.stderr], [], [])

    # Try reading some data from each
    stdoutPiece = read_async(process.stdout)
    stderrPiece = read_async(process.stderr)

    if stdoutPiece:
        print stdoutPiece,
    if stderrPiece:
        print stderrPiece,

    stdout += stdoutPiece
    stderr += stderrPiece
    returnCode = process.poll()

    if returnCode != None:
        return (returnCode, stdout, stderr)

Note that fcntl is only available on Unix-like platforms, including Cygwin.

If you need it to work on Windows without Cygwin, it’s doable, but it’s much, much tougher. You’ll have to:

Answered By: Adam Rosenfield

Answer #2:

Combining this answer with this, the following code works for me:

import subprocess, sys
p = subprocess.Popen(args, stderr=sys.stdout.fileno(), stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ""):
 print line,
Answered By: Sundae

Answer #3:

When I tested it, it seemed readline() is blocking. However I was able to access stdout and stderr separately using threads. Code sample as follows:

import os
import sys
import subprocess
import threading

class printstd(threading.Thread):
    def __init__(self, std, printstring):
        self.std = std
        self.printstring = printstring
    def run(self):
        while True:
          line = self.std.readline()
          if line != '':
            print self.printstring, line.rstrip()

pythonfile = os.path.join(os.getcwd(), '')

process = subprocess.Popen([sys.executable,'-u',pythonfile], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

print 'Process ID:',

thread1 = printstd(process.stdout, 'stdout:')
thread2 = printstd(process.stderr, 'stderr:')


threads = []


for t in threads:

However, I am not certain that this is thread-safe.

Answered By: user1379351

Leave a Reply

Your email address will not be published.