Scalable Middleware with Scheduled Tasks?

We are designing a Middleware that accepts requests from a Client. This Middleware should be scalable and be behind a load-balancer.

Our Middleware's main job is to process the request and forward it external servers for additional processing.

One thing we need to consider is scheduled processing; for example, a specific request can be scheduled to be forwarded ONLY at a specific time in the future (ex. after 3 hours).

Because of this, it introduces need to Persist Scheduled Jobs; it is needed to handle scenario where 1 server goes down, then other servers can still pick it up.

We have considered this approach:

1. Use DB storage to persist Scheduled Jobs
2. Middleware has background job periodically to query DB if there are any pending Jobs that needs to be sent
3. Use some kind of DB concurrency to handle multiple Middleware instances possible looking at same scheduled jobs

However, I am thinking that this design might not be optimal? Particularly, in the recurring background job because it might seem that some processing is wasted (ex. if there is always contention of data in DB).

Edit: Another alternative solution

1. Middleware receives scheduled request
2. Middleware creates / pushes scheduled request to a message queue (with time delay)
3. Separate function hosted (ex. Azure Functions) receive from Message Queue
4. Function processes the scheduled request

It seems with this approach, it is more scalable?

https://stackoverflow.com/questions/67327173/scalable-middleware-with-scheduled-tasks April 30, 2021 at 09:45AM