Pub-Sub Message Queue

Design an in-memory publish-subscribe message broker, a simplified Kafka: producers publish to topics, consumers in consumer groups read messages, and offsets track progress. It's a hard LLD problem because it is mostly about concurrency, delivery guarantees, and clean separation between storage and consumption.

Requirements

Functional

• Create topics with a number of partitions.
• Producers publish messages (optional key) to a topic; messages with the same key go to the same partition (ordering per key).
• Consumers subscribe as part of a consumer group; each partition is consumed by exactly one consumer in a group, and every group gets every message.
• Consumers poll for messages and commit offsets after processing.
• At-least-once delivery: uncommitted messages are redelivered after consumer failure or restart.
• Message retention by time or size; consumers can reset offsets (replay).

Non-functional

• Thread-safe: many producers and consumers concurrently.
• Ordering guaranteed within a partition.
• Slow consumers must not block producers or other groups.
• Rebalancing when consumers join or leave a group.