Python hangs with subprocess

I am having some issues figuring out why my code is hanging. I cannot reproduce it with a simple code, but I made a sample code on what/how the flow works. Basically the main process opens a subprocess that writes to stdout/stderr. The main process iterates through the stdout and the stderr is read by another thread so it doesnt hang when it writes too much. The thread reads the stderr then writes it to a queue. The main process then checks if the queue has some data, if it does it will throw an exception.

My problem is sometimes, reproducible with certain scenarios (unfortunately I cant reproduce here -- since this is the simple/sample workflow and the real one is a bit more complicated), where it hangs on checking the error in err = read_q(q). It seems that the thread never hit the finally section queue.put(None). Whats a good way of preventing this from happening? I tried to make sure the subprocess is terminated with reader.proc.terminate() after iterating through stdout but it didnt seem to help. Is it possible that the queue is full and it has reached a deadlock? It doesnt seem to be writing anything though. I guess the q.get is just waiting forever? Should I use a timeout? Its weird though since it works like 95% of the time, but not really sure what it doing differently on the scenario where its hanging.

Heres the sample code (I am using Python 3.8.2 and Ubuntu 20.04):

main.py

from threading import Thread  from queue import Queue  import subprocess      class Reader():      def __init__(self):          self.proc = subprocess.Popen(['python3', "mysubprocess.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)        def __iter__(self):          return self        def __next__(self):          while True:              line = self.proc.stdout.readline()              if not line:                  raise StopIteration              return line      def write_q(pipe, queue):      try:          with pipe:              for line in iter(pipe.readline, b''):                  queue.put((pipe, line))      finally:          queue.put(None)  # theres a scenario this doesnt get executed      def read_q(q):      return "".join([line.decode('utf-8') for source, line in iter(q.get, None)])      def read_stderr():      q = Queue()      stderr_thread = Thread(target=write_q, args=[reader.proc.stderr, q], daemon=True)      stderr_thread.start()      return q      def read_data(reader):      for line in reader:          print("Main read: {}".format(line))      def check_error(q):      err = read_q(q)  # theres a scenario where it hangs here      if err:          raise Exception("Error found: {}".format(str(err)))      if __name__ == "__main__":      reader = Reader()      q = read_stderr()      read_data(reader)      check_error(q)  

mysubprocess.py

import sys      for x in range(10):      sys.stdout.writelines(["something: {}\n".format(x)])        # print error      if x == 7:          sys.stderr.writelines(["error at {}".format(x)])  

https://stackoverflow.com/questions/66130756/python-hangs-with-subprocess February 10, 2021 at 12:03PM