Can I use PyFlink together with PyTorch/Tensorflow/ScikitLearn/Xgboost/LightGBM?

I am exploring PyFlink and I wonder if it is possible to use PyFlink together with all these ML libs that ML engineers normally use: PyTorch, Tensorflow, Scikit Learn, Xgboost, LightGBM, etc.

According to this SO thread, PySpark cannot use Scikit Learn directly inside UDF because Scikit Learn algorithms are not implemented to be distributed, while Spark runs distributedly.

Given PyFlink is similar to PySpark, I guess the answer maybe "no". But I would love to double check, and to see what I need to do to make PyFlink able to define UDFs using these ML libs.

https://stackoverflow.com/questions/66631859/can-i-use-pyflink-together-with-pytorch-tensorflow-scikitlearn-xgboost-lightgbm March 15, 2021 at 11:04AM