I have a piece of code below that creates a few threads to perform a task, which works perfectly well on its own. However I’m struggling to understand why the print statements I call in my function do not execute until all threads complete and the
print 'finished' statement is called. I would expect them to be called as the thread executes. Is there any simple way to accomplish this, and why does this work this way in the first place?
def func(param): time.sleep(.25) print param*2 if __name__ == '__main__': print 'starting execution' launchTime = time.clock() params = range(10) pool=multiprocessing.Pool(processes=100) #use N processes to download the data _=pool.map(func,params) print 'finished'
This happens due to stdout buffering. You still can flush the buffers:
import sys print 'starting' sys.stdout.flush()
For python 3 you can now use the
flush param like that:
print('Your text', flush=True)
Having run into plenty of issues around this and garbled outputs (especially under Windows when adding colours to the output..), my solution has been to have an exclusive printing thread which consumes a queue
If this still doesn’t work, also add
flush=True to your print statement(s) as suggested by @Or Duan
Further, you may find the “most correct”, but a heavy-handed approach to displaying messages with threading is to use the
logging library which can wrap a queue (and write to many places asynchronously, including stdout) or write to a system-level queue (outside Python; availability depends greatly on OS support)
import threading from queue import Queue def display_worker(display_queue): while True: line = display_queue.get() if line is None: # simple termination logic, other sentinels can be used break print(line, flush=True) # remove flush if slow or using Python2 def some_other_worker(display_queue, other_args): # NOTE accepts queue reference as an argument, though it could be a global display_queue.put("something which should be printed from this thread") def main(): display_queue = Queue() # synchronizes console output screen_printing_thread = threading.Thread( target=display_worker, args=(display_queue,), ) screen_printing_thread.start() ### other logic ### display_queue.put(None) # end screen_printing_thread screen_printing_thread.stop()