Skip to main content

Messaging Infrastructure Evaluation

Current State​

Redis Streams via @hmp/messaging (KafkaService interface backed by Redis).

When to Migrate to Kafka/NATS​

Migrate if ANY of these thresholds are exceeded:

  • Sustained event throughput > 1,000 events/second
  • More than 5 independent consumer groups
  • Need for multi-day event replay (Redis Streams limited by memory)
  • Compliance requires guaranteed delivery audit trail with retention > 7 days

Monitoring​

  • Event throughput: hmp_events_published_total (Prometheus counter)
  • Consumer lag: hmp_events_consumer_lag (Prometheus gauge)
  • Redis memory: redis_used_memory_bytes (Redis INFO)

Candidates​

  1. Apache Kafka / AWS MSK — Best for >10K events/sec, strong ordering guarantees
  2. NATS JetStream — Lighter than Kafka, good for 1K-10K events/sec
  3. Keep Redis Streams — Fine for <1K events/sec (current scale)

Decision​

Keep Redis Streams for now. Re-evaluate when traffic demands or compliance requirements change.