Deploy availability and event delivery — measured behaviour

Measured 2026-09-18 against hmp-infrastructure@f40a379eb, hmp-shared-packages@cc0cea0, and the 17 backend service repos at origin/master. hmp-infrastructure/origin/master advanced to 13d82c53b mid-probe (AB#11110); git diff --stat f40a379eb..13d82c53b -- kubernetes/ helm/ ci-cd/argocd/ monitoring/alerts/ terraform/envs/prod/ is EMPTY, so every reading below is re-verified unchanged at 13d82c53b.

Every claim below is anchored to a symbol. Re-derive line numbers with git grep -n … origin/master — never from a shared checkout.

Tracked as AB#14565. This document REPLACES the "Argo Rollouts for canary deployments with metrics-driven promotion" claim that stood in the root CLAUDE.md until this date.

1. Does a PROD deploy cause downtime?

For the HTTP request path: no — but the property holds by arithmetic accident, not by declaration.

1.1 No Deployment declares a rollout strategy

git grep -nE "^\s+(strategy|rollingUpdate|maxSurge|maxUnavailable):" origin/master -- kubernetes/ helm/
# → empty

All 21 apps inherit the Kubernetes default RollingUpdate with maxUnavailable: 25%, maxSurge: 25%. Percentages resolve as: maxUnavailable rounds down, maxSurge rounds up.

Replicas	maxUnavailable	maxSurge	Serving during rollout
2 (most services)	0	1	full
3 (prod minReplicas: 3 — gateway, auth, property, reservation, frontdesk, payment)	0	1	full
4+ (reachable by HPA scale-out to maxReplicas: 20)	1+	1+	degraded

So today a rollout never removes a pod before its replacement is Ready. That stops being true the moment the HPA scales a service to 4. Pinning strategy.rollingUpdate.maxUnavailable: 0 makes the guarantee survive scale-out.

1.2 The drain path is correct

preStop: ["/bin/sh","-c","sleep 10"] → terminationGracePeriodSeconds: 60 → ALB deregistration_delay.timeout_seconds=30 → readiness gating. 15 of 17 services call app.enableShutdownHooks() in src/main.ts.

Exceptions: hmp-analytics-service and hmp-revenue-strategy never call enableShutdownHooks(), so Nest never fires onModuleDestroy, KafkaService.isShuttingDown is never set, poll loops are not drained and Redis connections are not quit() on SIGTERM. Not data loss — the PEL covers it — but it adds a ≥30s reclaim delay per rolling restart on the service that owns every materialized view.

1.3 There is no canary in PROD

kind: Rollout exists in exactly one file, helm/hmp-service/templates/rollout.yaml, referenced only by ci-cd/argocd/_archived-hmp-{dev,prod,staging}.yaml. That chart implements strategy.blueGreen, not canary (already filed as AB#6272).

The live PROD path is ApplicationSet keynova-prod-services → kubernetes/overlays/prod/<service>/ → plain Deployments. No canary, no metric-gated promotion, no automatic rollback. A bad image reaches 100% of traffic and stays there until a human intervenes.

1.4 The migration / code skew window is hours wide

Every service ships migrations-job.yaml annotated argocd.argoproj.io/hook: Sync with argocd.argoproj.io/sync-wave: "-1", so prisma migrate deploy runs to completion before the Deployment is patched. Old pods therefore execute against the new schema for the entire rollout.

Boot budgets (startupProbe.failureThreshold × periodSeconds: 10):

Service	failureThreshold	Budget
api-gateway, analytics, notification	360	60 min
channel	180	30 min
12 other backends	60	10 min
web-dashboard, admin-portal	30	5 min
booking-engine	20	3.3 min

api-gateway's own manifest comment: "subscribes to 30+ Kafka topics sequentially — ~75-90s per topic, total boot ~45-60 min", with progressDeadlineSeconds: 3900. With maxSurge: 1 the 3 replicas start sequentially, so the manifests budget for a gateway rollout measured in hours.

⚠ MEASURED 2026-09-18 — that budget is stale by two orders of magnitude, and the numbers above are a CEILING that reality does not come close to.

Live reading, kn-prod, n=49 running pods in hmp-apps, 0 restarts: median app boot 33s, max 35s, min 5s. Not 45–60 minutes.

Tier	readiness initialDelaySeconds	measured container-start → Ready
15 backend services	30	30–35s (median 34)
web-dashboard, admin-portal	10	17–18s
booking-engine	5	5s

Total created→Ready is 30–70s, of which scheduling is 0–1s and image pull 0–36s. The failureThreshold: 360 comment is dated 2026-05-14; the O(n²) XGROUP CREATE cold boot it describes was fixed 2026-05-16 in @hmp/messaging ("now completes in seconds instead of an hour"), and deployment.yaml was last touched 2026-05-20 (fb69a67c1) for unrelated graceful-shutdown work. The fix worked; the threshold was never revisited.

Two conclusions, both actionable:

1. The migration/code skew window is ~1–3 minutes per service, not hours. Three replicas at maxSurge: 1 and ~50s each ≈ 2.5 min. §1.4's risk is real but far smaller than the manifests imply — a DROP COLUMN breaks traffic for minutes, not an afternoon.
2. failureThreshold: 360 is ~103× the observed maximum. A genuinely wedged backend pod is not restarted for 60 minutes. Sizing it to 18 (3 min) still leaves 5× headroom over the worst pod measured. This is now the larger of the two risks in this section.

And a third finding the measurement produced on its own: the 30–35s floor is not work, it is readinessProbe.initialDelaySeconds: 30. The correlation is exact — 30 → 30-35s, 10 → 17-18s, 5 → 5s. The startupProbe has no initialDelaySeconds, so it first fires at 10s and succeeds there; the pod then waits out the readiness delay doing nothing. With a startupProbe present that delay is redundant. Removing it takes every backend pod from ~33s to ~10–15s of time-to-serve — roughly a minute off every 3-replica rollout, and the same minute off every HPA scale-out response.

The job header reasons only about a failed migration ("Job stays Failed → ArgoCD waits → Deployment never updates → old pods continue serving") — correct, and a genuinely good property. The uncovered case is the successful non-backward-compatible migration: a DROP COLUMN, a RENAME, a new NOT NULL. That breaks live traffic for the whole window and reverting the image does not fix it. Expand/contract is mandatory here, not advisory.

1.5 PodDisruptionBudgets are the wrong shape

All 20 PDBs are minAvailable: 1. On a 3-replica service that permits a node drain to evict 2 of 3 simultaneously; on a 2-replica service it permits evicting 1 while a rollout is already down a pod. maxUnavailable: 1 expresses the intent at every replica count.

1.6 A sync-policy contradiction worth confirming live

ci-cd/argocd/keynova-prod.yaml carries the header comment "MANUAL sync for new git revisions per spec section 6 — prod requires human approval before new code is rolled out", but the spec sets:

syncPolicy:
  automated:
    prune: false
    selfHeal: true

with no syncWindows anywhere in ci-cd/argocd/. In ArgoCD the presence of automated enables auto-sync of new revisions; selfHeal is an additional drift behaviour, not a restriction. The FU-60a comment reads as a misunderstanding of that semantic.

CONFIRMED LIVE 2026-09-18. kubectl --context kn-prod get applications -n argocd reports automated=true, selfHeal=true on all 20 prod service Applications, with no syncWindows anywhere. The header comment is wrong: prod auto-syncs.

The real human gate is upstream — the ADO promotion approval before the bot commits the image tag — so image promotions are gated. But any other push under kubernetes/overlays/prod/** (a ConfigMap value, a replica count, a probe setting) reaches PROD on the next reconcile with no gate, and so does a kubernetes/base/apps/** edit, because every overlay renders from it. Treat a base manifest edit as a production change.

Also observed in the same reading: the keynova-prod ApplicationSet parent is OutOfSync with prune: true. Unrelated to this document, but worth an owner.

1.7 Data tier

terraform/envs/prod/main.tf: RDS multi_az = true, ElastiCache automatic_failover_enabled = true + multi_az_enabled = true. Failover is a 30–120s connection-level interruption, not a deploy concern, but it is the one event that reliably trips the best-effort publish path in §2.3.

---

2. Cross-service sync — delay and guarantee

Transport is Redis Streams. @hmp/messaging's class is named KafkaService for call-site compatibility only — there is no Kafka. Streams are keyed evt:<topic>; dead letters dlq:<topic>.

2.1 Latency: two numbers, and the SLO only measures one

Path A — direct kafkaService.publish() (195 call sites). XADD on a dedicated publisher connection (sub-ms) → consumer on its own connection per subscription (AB#9278) running XREADGROUP … COUNT 10 BLOCK 5000. A blocked consumer wakes on arrival, so steady-state delivery is network RTT — single-digit milliseconds.

The tail is handler-bound: each poll iteration runs reclaimPendingMessages (an XPENDING round-trip) before the read, and a batch is processed serially (for … await processMessage). One slow handler delays every later event on that topic for that consumer group.

Path B — transactional outbox (49 call sites). The row is written inside the DB transaction; the relay runs @Interval(5000) in all 14 services that have one. That adds 0–5s (≈2.5s median) before the XADD, then Path A on top.

The propagation metric cannot see Path B's wait. propagationSeconds(entryId, nowMs) parses the millisecond component of the Redis stream entry ID — i.e. XADD time. The time the event spent in the outbox table is invisible to the histogram. So outbox-routed events routinely exceed the declared SLO while the SLO reports green.

Declared SLO: HMPMessagingPropagationLatencyHigh in monitoring/alerts/messaging-alerts.yaml fires at p95 > 2s for 5m.

Retry cadence. A throwing handler does not XACK; the message sits in the PEL. Reclaim only touches entries with idleTime >= CLAIM_MIN_IDLE_MS (30s), and deliveryCount >= MAX_RETRIES (3) sends it to the DLQ. A flaky handler therefore retries at ~30s intervals and dead-letters after ~90s. That is the true worst-case sync delay.

2.2 Guarantee: at-least-once on consume

Consumer group + PEL + XCLAIM reclaim + DLQ, with XACK only after the handler returns. This half is sound.

2.3 Guarantee: best-effort on publish — no guarantee for ~80% of sites

publish() and publishMessage() never throw:

if (!this.isConnected) {
  this.publishFailCount++;
  this.logger.warn(`Cannot publish to ${topic}: event bus not connected …`);
  this.metricsService?.messagingEventsPublishFailedTotal?.inc({ service: svcName, topic });
  return;                    // ← business tx already committed. Event gone.
}

The catch around XADD does the same. A Redis blip or an ElastiCache failover means the Postgres write commits and the event is silently dropped, with no compensation.

Publish route	Sites
Direct publish() / publishMessage() — fire-and-forget	195
writeOutboxRow() / outbox.create() — transactionally durable	49

hmp-property-service is the worst case for durability: 20 direct publishes and 0 outbox rows. Rate, room-type and property changes propagate with no durability at all.

⚠ CORRECTION 2026-09-18. An earlier revision of this section also claimed property-service was "the only service whose app module omits MetricsModule", and that its @Optional() metricsService was therefore undefined. That was wrong. The per-service check read src/app.module.ts, where the real module is src/app/app.module.ts. MetricsModule is imported and registered at hmp-property-service/src/app/app.module.ts:93 under AB#14556, and the running prod image prod-10a9b64d… is origin/master HEAD — so it is live.

⚠ CORRECTION TO THIS CORRECTION — 2026-09-21. The paragraph above says MetricsModule "is imported and registered at hmp-property-service/src/app/app.module.ts:93". That reading came from the working tree, which sits on another session's unpushed branch feat/AB14557-mount-metrics-module. On origin/master property-service imports MessagingModule and no MetricsModule at all. So the original sweep's verdict was right about master even though it read the wrong file path, and the correction was wrong because it read the wrong ref.

Confirmed by measurement, not argument: after the fleet bump, kn-test showed property-service still logging No provider bound to MESSAGING_METRICS — one of three failures out of sixteen. AB#14557 is the card that mounts it.

The lesson is the one this document keeps relearning: cite the ref, not just the path. git show origin/master:<file>, never the working tree.

What is separately measured and true: property-service's three prod scrape targets are DOWN, together with asset-service's two (5 of 49), so its metrics do not reach Prometheus regardless of being registered. Tracked in AB#14568.

2.4 Three structural hazards in the consumer-group design

Every replica of a service shares one Redis consumer name. CONSUMER_NAME = consumer-${process.pid}, and every Dockerfile is CMD ["node", "dist/bundle.js"] with no init wrapper — node is PID 1, so every pod of every service registers as consumer-1. The happy path survives (XREADGROUP > still hands distinct entries to each caller), but the PEL is shared under that one name: pod A's reclaim loop sees pod B's in-flight messages and will XCLAIM and reprocess anything B has held longer than 30s. Any handler that can exceed 30s is exposed to concurrent duplicate processing across pods. Handler idempotency is load-bearing, not a nicety.

Consumer group names embed a process-local ordinal. consumerGroup = hmp-${SERVICE_NAME}-${topic}-${subscriberId} with subscriberId = options.subscriberId ?? `sub${++subscriberSequence}` . When a caller omits subscriberId the group name depends on the order of subscribe() calls in that process. Reorder them — a refactor, a newly conditional module — and the group is renamed, a fresh group is created at start-id '0' (the default unless fromBeginning: false), and you get a full stream replay: up to MAXLEN 10,000, or 500,000 on folio. / payment. / gst. / auth. / tenant. / audit. topics. The old group's offset is orphaned at the same moment.

MAXLEN trimming is silent loss. Every publish is XADD … MAXLEN ~ <n>. A consumer group that is down or lagging longer than it takes to push MAXLEN entries loses the unconsumed remainder with no error anywhere. The DLQ has the same property at 100,000 — the HMPMessagingDLQGrowing alert text admits it. Combined with the fact that only 2 of 41 mv_* models have a backfill script (see analytics-event-flow.md), a dropped event is effectively permanent.

---

2b. Probe right-sizing — VERIFIED on kn-test (AB#14567)

Phase 1 landed on the test overlays only (9bd1ca6d7), deliberately leaving kubernetes/base/ untouched: §1.6 confirms prod auto-syncs, so a base edit is a production change.

Change: on the 17 backend services, remove readinessProbe.initialDelaySeconds (redundant — kubelet already suppresses readiness until the startup probe succeeds, and readiness is a deep check that returns 503 until Postgres and Redis answer) and set startupProbe.failureThreshold: 18 (180s — the startup probe hits the shallow /api/health/live and succeeded at ~10s on every pod measured).

	Before	After
kn-test, container-start → Ready	median 32s, max 36s (n=34)	median 16s, min 5s, max 49s (n=36, 17/17 converged)
api-gateway specifically	34s	34s
Restarts	—	0

Median time-to-serve halved, 32s → 16s. But the distribution changed shape, and that is the more interesting result: the old 30s readiness floor was masking per-service variation. With it gone the spread is 5s to 49s —

• genuinely fast, entirely gated by the old delay: asset 5s, crm 5s, revenue-strategy 6s
• genuinely slow, and not helped by this change: property 49s, analytics 48s, channel 39s, frontdesk 38s

Two corrections to earlier revisions of this section.

1. An earlier revision reported "median 10s, ~69% cut" from the first 4 pods to reconcile. The converged sample (n=36) gives median 16s, i.e. ~50%.
2. An earlier revision claimed failureThreshold: 18 leaves "5× headroom over the slowest pod measured", based on a 35s max. The converged max is 49s, so the real margin is 180s / 49s ≈ 3.7×. Still comfortable, but smaller than stated.

The 38–49s tail was measured while 17 services rolled at once, so part of it is contention — readiness is a deep check (SELECT 1 + Redis) and many pods hit RDS together. A quiet-cluster re-read is still owed. Note property-service tops the tail at 49s and is also the service OOMKilling on test (AB#14569).

Pre-existing issues observed during the rollout, both unrelated to this change and both confirmed to carry the OLD probe config:

• hmp-api-gateway had two Failed pods aged 21h (startupFT=360, readyDelay=30). The new pods came up Running normally.
• hmp-property-service is OOMKilling on kn-test — 5 kills across 3 pods on 2026-09-16 and 2026-09-17, against a 512Mi limit. It is the only service in the namespace doing so, and it is already this document's worst case for publish durability and telemetry (§2.3). Worth its own card.

Phase 2 promotes the same change to kubernetes/base/apps/ and touches the qa, staging and prod overlays — per the 2026-05-23 trap, ArgoCD can miss a base change until the overlay is touched.

2c. The messaging alerts cannot fire — MEASURED on prod (AB#14568, P1)

Read from prometheus-kube-prometheus-stack-prometheus-0 in the monitoring namespace (not hmp-monitoring — the root CLAUDE.md namespace list is wrong about this too), 2026-09-18.

All 10 messaging alert rules are loaded and every one reports health=ok state=inactive. Nine of them reference metrics that do not exist.

An alert whose metric has no series is inactive forever, and on the alert page it is indistinguishable from a healthy one. Prod carries 6 hmp_* metric names in total, of which exactly one is hmp_messaging_*.

Alert	Metric it needs	Present?
HMPMessagingConsumerLagHigh / Critical	hmp_messaging_consumer_lag	absent
HMPMessagingDLQNonEmpty / DLQGrowing	hmp_messaging_dlq_size	absent
HMPMessagingPELDepthHigh / Stuck	hmp_messaging_pel_depth	absent
HMPMessagingPropagationLatencyHigh	hmp_messaging_event_propagation_seconds_bucket	absent
HMPMessagingConsumerStalled	hmp_messaging_events_processed_total	absent
HMPMessagingPublishFailureSpike	hmp_messaging_events_publish_failed_total	absent
HMPMessagingEventFailureRateHigh	hmp_messaging_events_failed_total	present

This is not a scrape-coverage problem. Scraping is healthy — 44 targets up, 5 down, across 18 services in hmp-apps. The series are not being produced.

Consequences. The declared p95 ≤ 2s propagation SLO of §2.1 is not measured at all. The DLQ alert cannot fire — which is exactly the mechanism behind the silently dead-lettered OTA booking, still unfixed. Consumer lag, PEL depth and publish failures are all invisible.

And there are real failures hiding behind it right now. The one surviving metric records 21 event-processing failures on prod, on hmp-analytics-service, across both replicas:

• reservation.no_show — 6 + 11 = 17
• folio.transaction.voided — 3 + 1 = 4

Every failure that exhausted MAX_RETRIES = 3 went to dlq:<topic>, which nothing can observe. These are materialized-view consumers, so no-show and voided-folio-transaction analytics on prod are silently incomplete — and per §2.4 only 2 of 41 mv_* models have a backfill script. HMPMessagingEventFailureRateHigh is a rate over 5m, so accumulated historical failures never tripped it.

2d. Root cause of the missing metrics — module registration order (AB#14568)

MessagingModule is @Global() and provides KafkaService, whose metricsService is @Optional(). A global module's exports are only resolvable by modules instantiated after it. So when MessagingModule is listed before MetricsModule in an app module's imports[], KafkaService is constructed while MetricsService does not yet exist, the injection resolves to undefined, and every messaging metric in that service becomes a permanent no-op — including the 30-second probeGaugeMetrics() interval, which is only scheduled inside if (this.metricsService).

Order	Services	Emits hmp_messaging_* on prod?
MetricsModule first (correct)	analytics, banquet, revenue-strategy — 3	analytics: yes
MetricsModule after MessagingModule	asset, auth, channel, crm, frontdesk, guest, housekeeping, notification, payment, pos, property, reservation, spa — 13	none

The only service producing any hmp_messaging_* series in prod is hmp-analytics-service — one of the three with the correct order. hmp-api-gateway registers neither module in its app module.

This was diagnosed before and never swept. hmp-analytics-service/src/app.module.ts carries the comment "AB#9459 — MetricsModule (@Global) must load before domain modules so …". Three services were fixed; thirteen were not.

Fix: move MetricsModule above MessagingModule in the 13 app modules, and add a gate asserting the ordering so it cannot regress.

⚠ ROOT CAUSE CORRECTED — 2026-09-20

The module-registration-order diagnosis above is WRONG and is withdrawn. The real cause is simpler and more severe.

KafkaService declares constructor(@Optional() private readonly metricsService?: MessagingMetrics). MessagingMetrics is a bare TypeScript interface (kafka.service.ts:14), not a class. Interfaces do not exist at runtime, so with emitDecoratorMetadata: true the emitted design:paramtypes is Object. Nothing is registered under the Object token, there is no @Inject() on the parameter, and MessagingModule provides no metrics provider — so @Optional() yields undefined in every service, always, since the code was written.

Every metric call inside @hmp/messaging is a permanent no-op platform-wide: the publish counters, the processed/failed counters, the eventPropagationSeconds histogram, and probeGaugeMetrics() — whose 30-second interval is scheduled only inside if (this.metricsService) and has therefore never run anywhere.

Why the ordering theory looked right. The single hmp_messaging_* series on prod, hmp_messaging_events_failed_total, is emitted by analytics' own code — event-dispatcher.service.ts:401 calls this.metrics?.messagingEventsFailedTotal?.inc(...) where this.metrics is @Optional() private readonly metrics?: MetricsService (line 135) — a class token, which resolves correctly. Analytics happened to sit in the "correct order" group and to carry its own instrumentation. The correlation was coincidental.

This explains everything the ordering theory could not: why events_published_total is absent despite constant publishing; why consumer_lag / pel_depth / dlq_size are absent; and why exactly one series exists, from exactly one service, on that service's own topic set.

Implication: reordering the 13 app modules would have fixed nothing — and would have shipped as a fix. The correction is an injection token: export a runtime token from @hmp/messaging, decorate with @Optional() @Inject(TOKEN), and provide { provide: TOKEN, useExisting: MetricsService }.

Secondary, and not explained by ordering: probeGaugeMetrics() swallows every error in two bare catch {} blocks, so even in analytics — where metricsService is wired — consumer_lag and pel_depth emit nothing and fail silently. That needs its own fix.

Ruled out by measurement: metric naming (all nine names at metrics.service.ts:341-397 match the alert queries exactly); scrape auth (AB#11314's metrics-scrape-token landed mid-probe — targets went to 44 up / 5 down, and a re-measure several scrape intervals later still showed 6 hmp_* names); and a missing MetricsModule in property-service (it is registered, and its running image is origin/master HEAD).

2e. Probe right-sizing — staging result, and what it says about the tail

Phase 2 landed on qa + staging (5fc25f455); prod is deliberately excluded and needs explicit approval, since §1.6 establishes that a push to a prod overlay is the prod rollout.

Staging: 17/17 converged, n=37, min 8s, median 13s, max 39s, 0 restarts — including hmp-banquet-service and hmp-notification-service, the two lacking last-applied-configuration. That is independent confirmation on a second environment that value: 1 lands where a field deletion cannot.

The annotation pre-flight is identical in all three live environments: exactly hmp-banquet-service and hmp-notification-service are missing it (2 of 20 on both staging and prod).

On the unresolved 38–49s tail. Staging gives a useful cross-check: hmp-analytics-service reads 9s on staging against 48s on test, and staging's slowest is 39s (frontdesk) against test's 49s (property). That is strong evidence the test tail is environment-specific — contention, or a slow test-side dependency — rather than intrinsic to those services. It is still not a controlled measurement.

⚠ Method correction. An earlier revision promised a "quiet-cluster re-read" to settle this. That is void: Ready.lastTransitionTime − containerStartedAt is a historical value fixed at pod start, so re-reading the same pods returns the same number by construction. Only a fresh roll on an idle cluster can separate contention from intrinsic boot cost.

⚠ The qa half of that commit is inert — the QA environment does not exist (no EKS cluster, no RDS, zero EC2 in account 851236939056), while the repo still carries 22 qa overlays, an ArgoCD ApplicationSet and terraform/envs/qa. See AB#14576. The documented test → qa → staging → prod promotion path is in reality test → staging → prod.

Decisions arising from this analysis are recorded in ADR — Silent-failure defaults across the platform. The findings below are evidence; the ADR is what we intend to change and why.

The DLQ replay that §2 shows is missing is designed in DLQ replay — design (AB#14842).

3. Remediation backlog

Ordered by availability / reliability impact per unit of risk.

#	Change	Where	Impact
1	Pin strategy.rollingUpdate: {maxUnavailable: 0, maxSurge: 1}	21 base Deployments	Makes zero-downtime survive HPA scale-out
2	Drop readinessProbe.initialDelaySeconds on the 15 backends (redundant behind the startupProbe)	15 base Deployments	~20s off every pod's time-to-serve; ~1 min off every 3-replica rollout and every HPA scale-out
3	Right-size startupProbe.failureThreshold 360→18 / 60→18	21 base Deployments	Wedged-pod detection 60 min → 3 min, still 5× headroom over the worst measured pod
4	Gate non-backward-compatible migrations in CI (expand/contract)	pipeline	Closes the (now ~1-3 min) skew window
5	Make publish() fail loudly, or route direct sites through the outbox	@hmp/messaging + 195 sites	Ends silent event loss on Redis failover
6	Add MetricsModule to hmp-property-service already done in AB#14556 — instead, fix the 5 DOWN prod scrape targets (property ×3, asset ×2)	AB#14568	Their metrics never reach Prometheus
7	Derive CONSUMER_NAME from HOSTNAME, not process.pid	@hmp/messaging	Removes cross-pod PEL collision + duplicate reprocessing
8	Require an explicit subscriberId on every subscribe()	@hmp/messaging + call sites	Prevents a refactor triggering a 500k-event replay
9	PDBs to maxUnavailable: 1	20 PDBs	Correct intent at every replica count
10	Measure propagation from business-event time, not XADD	@hmp/messaging + observability	Puts the outbox wait inside the SLO it must satisfy
11	app.enableShutdownHooks() in analytics + revenue-strategy	2 lines	Graceful consumer drain on every deploy

4. Open questions requiring a live reading

1. The actual syncPolicy on the running PROD ArgoCD Applications. RESOLVED 2026-09-18 — automated=true, selfHeal=true on all 20. See §1.6. 2. Whether hmp_messaging_* series are genuinely present in PROD Prometheus. RESOLVED 2026-09-18 — they are not. 6 of 7 families are absent and 9 of 10 alerts cannot fire. See §2c and AB#14568. All three open questions are now closed by measurement.

3. Real pod boot time. RESOLVED 2026-09-18 — measured on kn-prod, n=49 pods, 0 restarts: median 33s, max 35s. See §1.4.