Delayed Message Processing with Redis Streams
Speaker(s): Kurt Schoener
Delayed message processing can be valuable in many business processes. The problem of processing dynamically assigned delays is much more difficult than processing messages in a static processing window. For example, if we want to process payments between 2am and 4am, we can simply push messages onto a standard message queue system, wake the workers up at 2am and have them process the queue to depletion or until the 4am cutoff. We can scale this architecture easily—if the queue depth is too much for the current number of workers to deplete in the given window, we can simply add more workers. We can use any number of COTS products to implement this type of competitively processed workload in a static processing window.
Things become significantly more complex when you need to assign the delay dynamically based on rules for a specific message type or business process. You might, for example, delay order processing 15 minutes to give a customer the chance to modify the order without having to process a cancellation and then the creation of a new order. In our case, we need to delay customer communications associated with certain business processes in order to consolidate messaging for a more enjoyable customer experience. When a customer is approved for a lease, we would like them to have an email that includes some welcome language and the details of the approval. When the customer selects their merchandise, we want them to have an email with a link to their signing session, in case they need to sign later. The user interface walks the customer through each of these steps, and if they complete them quickly, then there is no need for a separate approval notice, and signing notice, etc. Instead, we would like to send a single consolidated email when the customer either completes the entire process or when they have not completed the next step for more than 30 minutes. Some of the complexities that needed to be addressed include: 1) How do you dynamically delay processing? 2) How can a later message cancel processing of an earlier message? 3) How do you accomplish this with a scalable, multi-worker architecture? 4) How do you manage messages that need to be processed immediately as well?
All the custom implementations solving dynamic delayed message processing with Redis since Redis in Action was published in 2013 have used some combination of lists and sorted sets. We were excited for the elegant solution that presented itself when streams became available in Redis 5. We have developed a streams-based solution to the problem that takes advantage of consumer groups for easily managing competitive processing. Due to a design that allows the consumers to be self-configuring, we have been able to develop a low-code solution, which simplifies it implementation and maintenance in our services-based architecture. We will describe and demonstrate our solution.