Built a single-node log system with Kafka-like semantics. Messages get appended with monotonic offsets, and clients can read from specific positions.
What it Does
- Send: Append messages to the log with auto-generated offsets
- Poll: Read messages from a given offset onwards
- Commit Offsets: Track consumer progress
- List Committed Offsets: Show current consumer positions
The Implementation
- Append-only log per key
- Monotonic offset generation (starting from 1)
- Consumer offset tracking
- Simple in-memory storage for single node
Key Design Choices
- Offset per Key: Each key has its own offset sequence
- Monotonic Offsets: Start at 1, increment sequentially
- Consumer Tracking: Map consumer -> key -> offset
- Append-Only: No message updates or deletes
Log Operations
- Send: Append message, return new offset
- Poll: Read messages from offset onwards
- Commit: Update consumer's position for a key
- List Commits: Show all consumer positions
Why This Works
- Simple append-only semantics
- Clear offset ordering
- Consumer progress tracking
- Foundation for multi-node scaling
This is the building block for distributed logs - once we have solid single-node semantics, we can replicate across multiple nodes.
