Event Based Kafka + Scheduling Design

Question

In an event based e commerce system using kafka, how do you handle auto expiring of orders in pending_payment status?

Typically, whenever there's a request from customer to create a new order there will be a message sent to topic so that inventory service can consume and reserve items the customer's order. Order s status will transition to pending_payment. For every order in pending_payment status, the items in order will be reserved for 30 minutes. I want to auto expire the orders and rollback the item's quantity in a scalable way.

One way i can think of is to schedule a cron job in 30 minutes that will expire a particular order and put back the item's quantity, but is there a more efficient way to do this? Is creating a cron job for every order a feasible way ?

Answer 1

there's no delayed queue functionality in kafka. there's also no way to "plug into" any broker-side time retention (and there's also no way to guarantee time retention processing happens when you want it to - think of it as GC).

so the general answer to your question is you're going to need a scheduler of some sort.

Answer 2

You need a way to manage a state - depending on time. Creating a cron would work but would arguably be difficult to maintain, monitor and scale, in particular as soon as you need more than one server to handle them.

Another solution is to use use a simple to use workflow engine such as zenaton (I'm cofounder). Typically you would launch a simple workflow written in your own programming language for each order. If a 'PaymentEvent' is not received by this workflow within 30 minutes, you decide to contact the inventory service to release the items.

Using such a tool, will provide you also a lot of flexibility to handle more complex situations, eg. what to do if payment arrives after 30 minutes