DLQ replay — design (AB#14842)

Status: Proposed. Evidence: deploy-availability-and-event-delivery.md, AB#14842, AB#14568 (the metrics that made this visible), AB#14628 (the gauge fix that lets a drained DLQ clear its alert).

The problem, stated precisely

A dead letter in dlq:<topic> is terminal. Nothing re-drives it when the condition that blocked it resolves.

This is not a defect in the handler. handleReservationModified rethrows P2025 deliberately (AB#10602) so a modify consumed before its reservation.created parent is redelivered rather than silently ACKed — the old swallow-and-ACK lost a sharer 4/4 on test. The DLQ is the designed terminal state, bounded so a genuinely orphaned modify cannot wedge the stream.

What fails is the budget against reality: redelivery is MAX_RETRIES (3) at CLAIM_MIN_IDLE_MS (30 s) ≈ 90 seconds. Measured on prod 2026-09-04: the parent row for e0fcd349… was created at 10:37:55Z against a modify at 05:41:15Z — a five-hour lag. No budget of that shape could have covered it. The event correctly dead-lettered, and then stayed dead after the blocker cleared.

Two ways the obvious design is wrong

1. Do not re-publish to evt:<topic>. XADD fans out to every consumer group on the stream. reservation.modified carries 22 groups; 10 of them processed the event successfully. Re-publishing converts one stuck event into 20+ duplicate deliveries, and this platform's handler idempotency is load-bearing but not guaranteed. The existing channel-service replay (AP-T11-016) does exactly this — acceptable there because its DLQ is a Postgres table with a single consumer, but not transferable here.

2. Do not replay unconditionally. 40fc9273… is still absent from the read-model. Replaying it re-fails, re-dead-letters, and churns. A replay must be gated on the blocking condition having actually resolved.

Design

Ownership: the consuming service, not a central sweeper

Only the consuming service knows two things a sweeper cannot: which consumerGroup is its own (the DLQ entry records it), and whether the blocker resolved (that question is service-specific — here, "does this reservation exist in my read-model yet?").

Mechanism: invoke the handler, never the stream

Replay calls the handler directly with the stored payload. No XADD, so no fan-out, no duplicate delivery to the groups that already succeeded. The service owns both the DLQ entry and the handler, so it can do this without any new transport.

Shape

A generic primitive in @hmp/messaging, wired per service — mirroring how OutboxRelayService is generic and each service provides its own model and schedule.

replayDlq({
  topic,                       // dlq:<topic> to scan
  ownsGroup: (g) => boolean,   // is this entry mine? (matches on consumerGroup)
  canReplay: async (payload) => boolean,   // has the blocker resolved?
  handler:   async (payload) => void,      // the SAME handler that failed
  maxAttempts, maxAgeDays,
})

Lifecycle, and where the safety lives

1. Scan dlq:<topic>; skip entries whose consumerGroup is not ours.
2. canReplay(payload) false → leave it, try again next cycle. No attempt is counted: a precondition that is not yet met is not a failure.
3. canReplay true → invoke the handler.
   • success → XDEL the entry. Only then.
   • throw → leave the entry, increment attempts, record the error.
4. attempts >= maxAttempts or age >= maxAgeDays → stop trying and surface it. An entry that can never be replayed must become visible, not retried forever.

XDEL-after-success is the property that matters: a replay that fails loses nothing, because the entry is still in the DLQ.

Concurrency

Multiple replicas will scan the same DLQ. Borrow the pattern channel-service already proved: claim before acting, release on failure, so two replicas cannot both drive one entry. A Redis SET NX on dlqreplay:<topic>:<id> with a short TTL is sufficient — the claim need only outlive one handler invocation.

Idempotency is still required

A handler may have partially applied before throwing. Replay therefore demands the same idempotency the reclaim path already demands (@hmp/messaging redelivers on any non-ACK). This design does not weaken that requirement; it inherits it.

Why this is observable now, and was not before

hmp_messaging_dlq_size did not exist until AB#14568 — the metric was a no-op in every service. And until AB#14628 a gauge label set was never removed, so a drained DLQ would have kept reporting its old depth forever and HMPMessagingDLQNonEmpty could never have cleared. Both had to land before a replay was worth building: without them, draining the queue would have been invisible and the alert would have stayed red.

Scope of the first implementation

reservation.modified on hmp-frontdesk-service, with canReplay = (p) => reservation row exists for p.reservationId. That is the measured case: 2 reservations, 5 events, one now replayable (e0fcd349…, parent present since 2026-09-04T10:37) and one correctly not (40fc9273…, still absent).

Explicitly out of scope: the 8 cohort-A dead letters from June, whose consumer groups were destroyed by the AB#14628 sweep. Nothing owns them, so no service can claim them — they need a one-off decision (replay under a current group, or discard with a reason), not an automated path.

Acceptance criteria

1. GIVEN a dead letter whose blocker has resolved, WHEN the replay runs, THEN the handler is invoked and the entry is XDELed only after it returns successfully.
2. GIVEN a dead letter whose blocker has NOT resolved, THEN it is left untouched and no attempt is counted.
3. GIVEN a replay, THEN no message is published to evt:<topic> — proven by asserting the other consumer groups' last-delivered-id is unchanged.
4. GIVEN an entry belonging to another service's consumer group, THEN it is skipped.
5. GIVEN two replicas replaying concurrently, THEN each entry is driven at most once.
6. GIVEN an entry past maxAttempts or maxAgeDays, THEN it is surfaced, not retried.
7. GIVEN a drained DLQ, THEN hmp_messaging_dlq_size falls to 0 and HMPMessagingDLQNonEmpty clears — the end-to-end proof that the loop closes.