Python C program subprocess hangs at “for line in iter”

Posted on

Solving problem is about exposing yourself to as many situations as possible like Python C program subprocess hangs at “for line in iter” and practice these strategies over and over. With time, it becomes second nature and a natural way you approach any problems in general. Big or small, always start with a plan, use other strategies mentioned here till you are confident and ready to code the solution.
In this post, my aim is to share an overview the topic about Python C program subprocess hangs at “for line in iter”, which can be followed any time. Take easy to follow this discuss.

Python C program subprocess hangs at “for line in iter”

Ok so I’m trying to run a C program from a python script. Currently I’m using a test C program:

#include <stdio.h>
int main() {
    while (1) {
        printf("2000n");
        sleep(1);
    }
    return 0;
}

To simulate the program that I will be using, which takes readings from a sensor constantly.
Then I’m trying to read the output (in this case "2000") from the C program with subprocess in python:

#!usr/bin/python
import subprocess
process = subprocess.Popen("./main", stdout=subprocess.PIPE)
while True:
    for line in iter(process.stdout.readline, ''):
            print line,

but this is not working. From using print statements, it runs the .Popen line then waits at for line in iter(process.stdout.readline, ''):, until I press Ctrl-C.

Why is this? This is exactly what most examples that I’ve seen have as their code, and yet it does not read the file.

Is there a way of making it run only when there is something to be read?

Answer #1:

It is a block buffering issue.

What follows is an extended for your case version of my answer to Python: read streaming input from subprocess.communicate() question.

Fix stdout buffer in C program directly

stdio-based programs as a rule are line buffered if they are running interactively in a terminal and block buffered when their stdout is redirected to a pipe. In the latter case, you won’t see new lines until the buffer overflows or flushed.

To avoid calling fflush() after each printf() call, you could force line buffered output by calling in a C program at the very beginning:

setvbuf(stdout, (char *) NULL, _IOLBF, 0); /* make line buffered stdout */

As soon as a newline is printed the buffer is flushed in this case.

Or fix it without modifying the source of C program

There is stdbuf utility that allows you to change buffering type without modifying the source code e.g.:

from subprocess import Popen, PIPE
process = Popen(["stdbuf", "-oL", "./main"], stdout=PIPE, bufsize=1)
for line in iter(process.stdout.readline, b''):
    print line,
process.communicate() # close process' stream, wait for it to exit

There are also other utilities available, see Turn off buffering in pipe.

Or use pseudo-TTY

To trick the subprocess into thinking that it is running interactively, you could use pexpect module or its analogs, for code examples that use pexpect and pty modules, see Python subprocess readlines() hangs. Here’s a variation on the pty example provided there (it should work on Linux):

#!/usr/bin/env python
import os
import pty
import sys
from select import select
from subprocess import Popen, STDOUT
master_fd, slave_fd = pty.openpty()  # provide tty to enable line buffering
process = Popen("./main", stdin=slave_fd, stdout=slave_fd, stderr=STDOUT,
                bufsize=0, close_fds=True)
timeout = .1 # ugly but otherwise `select` blocks on process' exit
# code is similar to _copy() from pty.py
with os.fdopen(master_fd, 'r+b', 0) as master:
    input_fds = [master, sys.stdin]
    while True:
        fds = select(input_fds, [], [], timeout)[0]
        if master in fds: # subprocess' output is ready
            data = os.read(master_fd, 512) # <-- doesn't block, may return less
            if not data: # EOF
                input_fds.remove(master)
            else:
                os.write(sys.stdout.fileno(), data) # copy to our stdout
        if sys.stdin in fds: # got user input
            data = os.read(sys.stdin.fileno(), 512)
            if not data:
                input_fds.remove(sys.stdin)
            else:
                master.write(data) # copy it to subprocess' stdin
        if not fds: # timeout in select()
            if process.poll() is not None: # subprocess ended
                # and no output is buffered <-- timeout + dead subprocess
                assert not select([master], [], [], 0)[0] # race is possible
                os.close(slave_fd) # subproces don't need it anymore
                break
rc = process.wait()
print("subprocess exited with status %d" % rc)

Or use pty via pexpect

pexpect wraps pty handling into higher level interface:

#!/usr/bin/env python
import pexpect
child = pexpect.spawn("/.main")
for line in child:
    print line,
child.close()

Q: Why not just use a pipe (popen())? explains why pseudo-TTY is useful.

Answered By: jfs

Answer #2:

Your program isn’t hung, it just runs very slowly. Your program is using buffered output; the "2000n" data is not being written to stdout immediately, but will eventually make it. In your case, it might take BUFSIZ/strlen("2000n") seconds (probably 1638 seconds) to complete.

After this line:

printf("2000n");

add

fflush(stdout);
Answered By: Rob?

Answer #3:

See readline docs.

Your code:

process.stdout.readline

Is waiting for EOF or a newline.

I cannot tell what you are ultimately trying to do, but adding a newline to your printf, e.g., printf("2000n");, should at least get you started.

Answered By: codnodder
The answers/resolutions are collected from stackoverflow, are licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0 .

Leave a Reply

Your email address will not be published.