Fallout: Dead Letter Management At Scale
Joris Kuipers - Trifork
Many Spring apps use point-to-point messaging via one of the supported protocols or APIs: JMS, AMQP, AWS SQS, etc. Message-Oriented Middleware typically provides dead lettering (DL) support to deal with messages that cannot be processed, moving them to a dedicated queue to prevent them from becoming poison pills. But what do you do when messages end up on a DLQ? And what if this happens as part of a platform that processes millions of messages each month for dozens of use cases? In this talk Joris will explain how his team developed a DLQ management solution for their AWS-based integration platform with Spring. It allows for monitoring all DLQs, automating retries by periodically moving DL-ed messages back to their source queue while tracking how often this has happened, and for inspecting not only the payload of messages but all related logging as well. Messages can also be pruned, individually or per DLQ, which may happen automatically after too many retries. You’ll learn about the importance of idempotency to design self-healing systems and tracing for correlating logs to understand why processing of a message failed. Next to explaining these concepts, Joris will cover the implementation (incl. gotchas like inspecting messages without consuming them) and provide a live demo of the system. This talk will leave you with a good understanding of what to consider when dealing with DL-ed messages and how you can automate that for your own systems!