catching stdout in realtime from subprocess

0 votes
asked Oct 22, 2009 by john-a

I want to subprocess.Popen() rsync.exe in Windows, and print the stdout in Python.

My code works, but it doesn't catch the progress until a file transfer is done! I want to print the progress for each file in real time.

Using Python 3.1 now since I heard it should be better at handling IO.

import subprocess, time, os, sys

cmd = "rsync.exe -vaz -P source/ dest/"
p, line = True, 'start'


p = subprocess.Popen(cmd,
                     shell=True,
                     bufsize=64,
                     stdin=subprocess.PIPE,
                     stderr=subprocess.PIPE,
                     stdout=subprocess.PIPE)

for line in p.stdout:
    print(">>> " + str(line.rstrip()))
    p.stdout.flush()

11 Answers

0 votes
answered Jan 22, 2009 by will

Change the stdout from the rsync process to be unbuffered.

p = subprocess.Popen(cmd,
                     shell=True,
                     bufsize=0,  # 0=unbuffered, 1=line-buffered, else buffer-size
                     stdin=subprocess.PIPE,
                     stderr=subprocess.PIPE,
                     stdout=subprocess.PIPE)
0 votes
answered Oct 22, 2009 by nosklo

Some rules of thumb for subprocess.

  • Never use shell=True. It needlessly invokes an extra shell process to call your program.
  • When calling processes, arguments are passed around as lists. sys.argv in python is a list, and so is argv in C. So you pass a list to Popen to call subprocesses, not a string.
  • Don't redirect stderr to a PIPE when you're not reading it.
  • Don't redirect stdin when you're not writing to it.

Example:

import subprocess, time, os, sys
cmd = ["rsync.exe", "-vaz", "-P", "source/" ,"dest/"]

p = subprocess.Popen(cmd,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.STDOUT)

for line in iter(p.stdout.readline, b''):
    print(">>> " + line.rstrip())

That said, it is probable that rsync buffers its output when it detects that it is connected to a pipe instead of a terminal. This is the default behavior - when connected to a pipe, programs must explicitly flush stdout for realtime results, otherwise standard C library will buffer.

To test for that, try running this instead:

cmd = [sys.executable, 'test_out.py']

and create a test_out.py file with the contents:

import sys
import time
print ("Hello")
sys.stdout.flush()
time.sleep(10)
print ("World")

Executing that subprocess should give you "Hello" and wait 10 seconds before giving "World". If that happens with the python code above and not with rsync, that means rsync itself is buffering output, so you are out of luck.

A solution would be to connect direct to a pty, using something like pexpect.

0 votes
answered Jan 9, 2011 by mikegm

I've noticed that there is no mention of using a temporary file as intermediate. The following gets around the buffering issues by outputting to a temporary file and allows you to parse the data coming from rsync without connecting to a pty. I tested the following on a linux box, and the output of rsync tends to differ across platforms, so the regular expressions to parse the output may vary:

import subprocess, time, tempfile, re

pipe_output, file_name = tempfile.TemporaryFile()
cmd = ["rsync", "-vaz", "-P", "/src/" ,"/dest"]

p = subprocess.Popen(cmd, stdout=pipe_output, 
                     stderr=subprocess.STDOUT)
while p.poll() is None:
    # p.poll() returns None while the program is still running
    # sleep for 1 second
    time.sleep(1)
    last_line =  open(file_name).readlines()
    # it's possible that it hasn't output yet, so continue
    if len(last_line) == 0: continue
    last_line = last_line[-1]
    # Matching to "[bytes downloaded]  number%  [speed] number:number:number"
    match_it = re.match(".* ([0-9]*)%.* ([0-9]*:[0-9]*:[0-9]*).*", last_line)
    if not match_it: continue
    # in this case, the percentage is stored in match_it.group(1), 
    # time in match_it.group(2).  We could do something with it here...
0 votes
answered Oct 10, 2012 by erik

You cannot get stdout to print unbuffered to a pipe (unless you can rewrite the program that prints to stdout), so here is my solution:

Redirect stdout to sterr, which is not buffered. '<cmd> 1>&2' should do it. Open the process as follows: myproc = subprocess.Popen('<cmd> 1>&2', stderr=subprocess.PIPE)
You cannot distinguish from stdout or stderr, but you get all output immediately.

Hope this helps anyone tackling this problem.

0 votes
answered Jan 2, 2013 by zviadm

Your problem is:

for line in p.stdout:
    print(">>> " + str(line.rstrip()))
    p.stdout.flush()

the iterator itself has extra buffering.

Try doing like this:

while True:
  line = p.stdout.readline()
  if not line:
     break
  print line
0 votes
answered Oct 12, 2013 by ibue
for line in p.stdout:
  ...

always blocks until the next line-feed.

For "real-time" behaviour you have to do something like this:

while True:
  inchar = p.stdout.read(1)
  if inchar: #neither empty string nor None
    print(str(inchar), end='') #or end=None to flush immediately
  else:
    print('') #flush for implicit line-buffering
    break

The while-loop is left when the child process closes its stdout or exits. read()/read(-1) would block until the child process closed its stdout or exited.

0 votes
answered Jan 6, 2016 by ling

On Linux, I had the same problem of getting rid of the buffering. I finally used "stdbuf -o0" (or, unbuffer from expect) to get rid of the PIPE buffering.

proc = Popen(['stdbuf', '-o0'] + cmd, stdout=PIPE, stderr=PIPE)
stdout = proc.stdout

I could then use select.select on stdout.

See also https://unix.stackexchange.com/questions/25372/

0 votes
answered Jan 14, 2016 by nithin

To avoid caching of output you might wanna try pexpect,

child = pexpect.spawn(launchcmd,args,timeout=None)
while True:
    try:
        child.expect('\n')
        print(child.before)
    except pexpect.EOF:
        break

PS : I know this question is pretty old, still providing the solution which worked for me.

PPS: got this answer from another question

0 votes
answered Oct 3, 2016 by elvin

I know this is an old topic, but there is a solution now. Call the rsync with option --outbuf=L. Example:

cmd=['rsync', '-arzv','--backup','--outbuf=L','source/','dest']
p = subprocess.Popen(cmd,
                     stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, b''):
    print '>>> {}'.format(line.rstrip())
0 votes
answered Sep 15, 2017 by banghuazhao

Use | tee to redirect the stdout to a file named out.txt while displaying stdout in realtime on terminal

import subprocess, time, os, sys

cmd = "rsync.exe -vaz -P source/ dest/ | tee out.txt"

p, line = True, 'start'

p = subprocess.Popen(cmd,
                 shell=True)

p.wait()

You can get the stdout from the file out.txt after subprocess.

# Get stdout from file out.txt
f = open('out.txt')
out = f.read()
f.close()
Welcome to Q&A, where you can ask questions and receive answers from other members of the community.
Website Online Counter

...