Dask hangs when running within Celery

I need to add Dask functionality to an existing Flask + Celery application. I tried the code below, but after creating the Dask data frame from Pandas any Dask function hangs.

What is the problem? Are there any considerations I'm not taking into account?

Note that my objective is to use Celery to add multithreaded capabilities to Flask, and Dask to process data.

import pandas as pd  import dask  import dask.dataframe as dd  from ..celery_app import service    @service.task  def train_model(req):        pdf = pd.DataFrame({              'height': [6.21, 5.12, 5.85, 5.78, 5.98],              'weight': [150, 126, 133, 164, 203]          })        ddf = dd.from_pandas(pdf, npartitions=2)         print (len(ddf.index))   # <--- program hangs here      return { 'ret': 0 }  

https://stackoverflow.com/questions/66863986/dask-hangs-when-running-within-celery March 30, 2021 at 10:25AM