Better approach for python multitasking for task queue framework

I'm working on Task Queue framework for a web scraper. The framework will launch a multiple of workers/spiders and each will pool a task from the qeue. Now I'm having a difficulty to choose between different approach to multitasking. Which is the better in your opinion.

1. Multithreaded with AsyncIO (asycio.pool within each thread so whenever a task needed brief sleep, it can await so worker can process other task).

2. Multiprocess with AsyncIO (same as above, but avoid GIL).

3. Run multiple python process (each still have asyncio workers).

Thanks.

https://stackoverflow.com/questions/65622461/better-approach-for-python-multitasking-for-task-queue-framework January 08, 2021 at 10:04AM