Python process running in Docker hangs after ~2 hours, container is still running but there is no stdout, stderr or new data saved

I've got a docker-compose.yml which runs a Python process inside a Docker container as run using docker-compose up.

The python process is relatively simple, the pseudocode would be along the lines of:

while True:      logging.info('Start')      fetch_some_data()  # takes 5 mins to an hour to complete depending on the amount of new data      # Sleep for 1 hour      for minute in range(60):          logging.info(f'sleeing {minute}')          time.sleep(60)      logging.info('End')  

The issue is that the process tends to hang after a period of time (~2 hours observed). This can happen on inside fetch_some_data() or during time.sleep(60).

For example the docker container is still up and according to the logs it is between subtasks inside fetch_some_data() but nothing is happening: the python process is visible in top inside the container but there is no stdout or stderr and not data received.

I have also set ENV PYTHONUNBUFFERED=1 in the Dockerfile for unbuffered logging however even if logging was somehow buffered data should still be received, and it definitely isn't.

The host is running:

$ lsb_release -a  No LSB modules are available.  Distributor ID: Ubuntu  Description:    Ubuntu 20.04.2 LTS  Release:        20.04  Codename:       focal    $ docker -v  Docker version 20.10.5, build 55c4c88    $ docker-compose -v  docker-compose version 1.28.6, build 5db8d86f  

https://stackoverflow.com/questions/67394281/python-process-running-in-docker-hangs-after-2-hours-container-is-still-runnin May 05, 2021 at 10:05AM