python3 multiprocessing map_async not running all tasks

My code looks like this:

import multiprocessing as mp    def myfunc(args):      val1, val2 = args      print("now running", val1, val2, "node=", mp.current_process().name)      do_stuff()      return (args, output)    args = [(1,2), (2,3), (4,5), ...] # 27654 args    pool = mp.Pool(processes = 16)    gen_output = pool.map_async(myfunc, args)    for output in gen_output.get():       args_copy, output_i = output        

My initial list (args) has 27,654 tuples, but after counting the printout ("now running"), I only get 14,671 back. Also, I can't easily check, but it also seems I dont get output for the args that don't print out.
There is no error message or warning other than the printout of the args that did run. \

I suspect some child nodes didn't run at all. Does anyone know what may cause this and how to get around?
The printing statement should tell me which task is running on each child node. But oftern times it only prints 6 or 7 different children "ForkPoolWorker" 1,5,6,9,12,16 for example. \ I am using a SLURM cluster to run this script and request 16 nodes.

>>> cat job.sh  /bin/bash  #SBATCH -o tmp.out  #SBATCH -e tmp.err  #SBATCH -n 16   #SBATCH -p my_partition  #SBATCH --mem=16g    python3 my_script.py. -n 16   

argument n will go to pool = mp.Pool(processes = options.n)

https://stackoverflow.com/questions/66579577/python3-multiprocessing-map-async-not-running-all-tasks March 11, 2021 at 05:20PM