python multiprocessing start and close processes independently

I am trying to run inference with tensorflow using multiprocessing. Each process uses 1 GPU. I have a list of files input_files[]. Every process gets one file, runs the model.predict on it and writes the results to file. To move on to next file, I need to close the process and restart it. This is because tensorflow doesn't let go of memory. So if I use the same process, I get memory leak.

I have written a code below which works. I start 5 processes, close them and start another 5. The issue is that all processes need to wait for the slowest one before they can move on. How can I start and close each process independent of the others?

Note that Pool.map is over input_files_small not input_files.

file1 --> start new process --> run prediction --> close process --> file2 --> start new process --> etc.      for i in range(0, len(input_files), num_process):      input_files_small = input_files[i:i+num_process]      try:          process_pool = multiprocessing.Pool(processes=num_process, initializer=init_worker, initargs=(gpu_ids))          pool_output = process_pool.map(worker_fn, input_files_small)      finally:          process_pool.close()          process_pool.join()  

https://stackoverflow.com/questions/66880262/python-multiprocessing-start-and-close-processes-independently March 31, 2021 at 08:58AM