Achieving True Session Ordering in High-Throughput Kafka Pipelines with Go

Article Summary

This technical deep dive outlines the architectural challenges of maintaining strict message sequence for conversational AI when processing massive volumes of data through multi-stage pipelines (NLU, LLM inference, etc.). The core problem is that Kafka only guarantees ordering within a partition, which is insufficient when many sessions interleave within the same partition. The solution implemented involves a two-level worker hierarchy: Level 1 uses consistent hashing to route all messages from a single session to the same worker, and Level 2 assigns a dedicated goroutine to that session. This goroutine processes messages sequentially, ensuring order while allowing massive parallelism across different sessions. Furthermore, the design tackles failure modes by implementing in-place retries within the session goroutine, which naturally blocks subsequent messages until the current one resolves, and by using contiguous watermark commits to prevent data loss or skipping upon system crashes.

Key Points

• The architecture uses consistent hashing to ensure all messages belonging to a single user session are routed to the same dedicated worker for sequential processing.
• A two-level worker system employs a per-session goroutine to guarantee message order within a session, even while processing thousands of sessions in parallel.
• Crash recovery is achieved via contiguous watermark commits, ensuring offsets are only committed up to the highest sequence number that has been fully and sequentially processed.