How to properly implement producer consumer in python

I have two threads in a producer consumer pattern. When the consumer receives data it calls an time consuming function expensive() and then enters in a for loop.

But if while the consumer is working new data arrives, it should abort the current work, (exit the loop) and start with the new data.

I tried with a queue.Queue something like this:

q = queue.Queue()    def producer():      while True:          ...          q.put(d)          def consumer():      while True:          d = q.get()          expensive(d)          for i in range(10000):              ...              if not q.empty():                  break        

But the problem with this code is that if the producer put data too too fast, and the queue get to have many items, the consumer will do the expensive(d) call plus one loop iteration and then abort for each item, which is time consuming. The code should work, but is not optimized.

https://stackoverflow.com/questions/67013844/how-to-properly-implement-producer-consumer-in-python April 09, 2021 at 09:22AM