BrokerLite

A complete in-memory message broker built from scratch in Python. Zero external dependencies. Pub/sub, queues, consumer groups, WAL, retries, dead letters, TCP protocol — all stdlib.

22Core modules
315Tests passing
0Dependencies
~5kLines of code

Publish → Consume in 6 lines

from brokerlite.broker import Broker
from brokerlite.message import Message

broker = Broker()
broker.start()
broker.create_topic("orders")
broker.publish(Message(topic="orders", value=b"hello"))

Core Primitives

  • Partitioned topics with key routing
  • Consumer groups (range + round-robin)
  • FIFO & priority queues
  • Write-ahead log with segments
  • Ack tracking & dead letter queues
  • Configurable retry policies
  • Token-bucket rate limiting
  • Binary wire protocol
  • TCP server & client
  • Schema registry
  • Middleware pipeline
  • CLI admin tool
GitHub →

Pub/Sub

Topics split into partitions for parallelism. Messages route by key hash. Consumer groups rebalance automatically when members join or leave.

topicspartitionsconsumer-groups

Reliability

Every message is tracked. Failed deliveries retry with exponential backoff and jitter. After max attempts, messages land in a dead letter queue for inspection and replay.

ackretrydlq

Storage

Append-only write-ahead log with automatic segment rotation. SQLite metadata for committed offsets and topic configs. Key-based compaction and time-based retention.

WALSQLitecompaction

Architecture

Producer → Middleware → Topic ├── Partition 0 ├── Partition 1 └── Partition N ↓ ConsumerGroup ├── Consumer A └── Consumer B ↓ AckManager → Retry → DLQ

TCP Server & Client

# Start server
server = BrokerServer(host="0.0.0.0", port=9092)
server.start(background=True)

# Connect client
client = BrokerClient(host="localhost", port=9092)
client.connect()
client.produce("events", b"click", key="user-1")
msgs = client.fetch("events", partition=0, offset=0)