Skip to main content

AP-T11-008 — Webhook Observability (Prometheus + Alertmanager)

Status: OPEN — to be executed before production launch Owner: Channel-service team Audit code: AP-T11-008 Branch convention: audit/AP-T11-008-webhook-observability Spawned by: AP-T11-007 (ResAvenue cert single endpoint, 2026-04-30)

Background​

AP-T11-007 shipped the single inbound endpoint POST /api/webhooks/resavenue/booking and proved end-to-end on QA. During that work we discovered four latent bugs (JwtAuthGuard 401, missing XML parser, SanitizeInterceptor stripping XML, missing tenantId on inserts) — all of which had been silently broken on the legacy /reservation and /cancellation routes for over a month because no inbound webhook traffic had ever reached the service from outside. We had no way to know the routes were broken because we had no observability on webhook_events.

This ticket addresses that gap permanently.

Goal​

Make every inbound webhook event observable as a Prometheus counter, and surface the high-signal failure modes (auth rejection rate, parse errors, kafka-publish failures) on Alertmanager → Slack/PagerDuty using the existing routing in hmp-infrastructure/monitoring/alerts/.

This is not ResAvenue-specific — the same observability covers STAAH, eZee, ChannelRush, UrbanPiper, and any future intermediary plugged into AbstractIntermediaryWebhookController or WebhookEvent-writing controller.

Deliverables​

1. Counter (channel-service)​

Add a Prometheus counter to @hmp/observability (or to hmp-channel-service directly if @hmp/observability is too generic):

const webhookEventsTotal = new Counter({
  name: 'channel_webhook_events_total',
  help: 'Inbound webhook events received by channel-service, partitioned by provider, eventType, and status',
  labelNames: ['provider', 'eventType', 'status', 'tenantId'] as const,
});

Increment in AbstractIntermediaryWebhookController.persistWebhookEvent() after the DB insert. The existing tenantId fallback (sentinel UUID for unmatched payloads) flows through naturally.

2. Alertmanager rules (hmp-infrastructure)​

New file hmp-infrastructure/monitoring/alerts/hmp-channel-webhook-alerts.yaml:

AlertConditionSeverityRouting
WebhookAuthRejectionRaterate(channel_webhook_events_total{status=~"REJECTED_AUTH|REJECTED_IP|REJECTED_NO_SECRET"}[5m]) > 0.5warningSlack #ops
WebhookParseErrorRaterate(channel_webhook_events_total{status="PARSE_ERROR"}[5m]) > 0.1warningSlack #ops
WebhookKafkaPublishFailurerate(channel_webhook_events_total{status="RETRY_PENDING"}[10m]) > 0criticalPagerDuty
WebhookSilenceFromActivePartnerabsent(rate(channel_webhook_events_total{provider!=""}[6h])) and on() time() - <last_known_partner_activity_time> > 6hwarningSlack #ops
WebhookFirstInboundFromPartnerincrease(channel_webhook_events_total{provider="<X>", status="ACCEPTED"}[1h]) > 0 and last value 0infoSlack #ops

The last rule (WebhookFirstInboundFromPartner) is the cert-time alert — parameterised so we add a one-line entry per partner during cert and silence/remove after sign-off.

3. Grafana dashboard panel​

Add to hmp-infrastructure/monitoring/dashboards/hmp-channel-service.json:

  • Webhook events / 5min (stacked by status) — line chart
  • Webhook auth rejection rate by provider — line chart, last 24h
  • First-time provider seen — table, last 7 days

4. Documentation​

Update hmp-channel-service/CLAUDE.md "Inbound Webhooks" table with the metric name + alert names. Update hmp-docs/docs/integrations/resavenue-integration-guide.md Part 9 with the new WebhookFirstInboundFromPartner rule the cert team should silence after sign-off.

Effort estimate​

  • Counter + persist hook wiring: ~30 min
  • Alertmanager YAML + Grafana JSON: ~30 min
  • Doc updates: ~15 min
  • QA smoke (synthetic events through /booking should bump the counter): ~15 min

Total: ~90 min, single PR.

When to execute​

Before production launch. Tracking signal: any go/no-go review for production where ResAvenue or another intermediary is critical-path. The same review that flips CHANNEL_MODE=live should require this ticket green.

Definition of done​

  • Counter shipped + visible at :9464/metrics (channel-service)
  • Alertmanager rules merged + loaded by Prometheus
  • One synthetic test through /api/webhooks/resavenue/booking bumps the counter as expected
  • One synthetic auth-fail bumps REJECTED_AUTH and (if rate threshold reached) fires the alert
  • CLAUDE.md + integration guide updated
  • Cert-time WebhookFirstInboundFromPartner rule silenced/removed for ResAvenue