EP-020: Infrastructure & DevOps

Business Objective

A hospitality platform handling guest PII, payment data, and real-time operations (check-in, POS, housekeeping) demands production-grade infrastructure with high availability, automated deployments, observability, and disaster recovery. Downtime during peak check-in hours (2-6 PM) directly translates to guest dissatisfaction and revenue loss. Robust DevOps practices reduce mean time to recovery (MTTR), enable zero-downtime deployments, and ensure compliance with PCI-DSS and data residency requirements (India's DPDP Act mandates data stay within Indian borders for the primary region).

Audience

• Primary: DevOps Engineers, SRE Team, Platform Engineers
• Secondary: Backend Developers (CI/CD consumers), Engineering Managers (cost/uptime dashboards), Security/Compliance Officers

Scope

In Scope

• Docker deployment configurations (demo stack with 11 containers, dev environment)
• Kubernetes orchestration with Kustomize (base, overlays, addons) and Helm charts (per-service)
• Terraform IaC modules: networking (VPC, subnets, NAT), compute (EKS), database (RDS PostgreSQL), cache (ElastiCache Redis), storage (S3 + CloudFront CDN), WAF, multi-region (Route53 failover, S3 replication, secondary EKS)
• CI/CD pipeline: GitHub Actions (lint, test, build Docker images) -> ArgoCD (GitOps deploy to Kubernetes)
• Monitoring stack: Prometheus metrics collection, Grafana dashboards, Jaeger distributed tracing, Loki log aggregation, OpenTelemetry Collector
• Alerting: Prometheus alert rules (SLA-based), PagerDuty integration
• Load testing: k6 scripts for 8 service scenarios (auth, reservations, booking-engine, frontdesk, housekeeping, POS, channel, full-load)
• Backup and recovery: RDS automated snapshots, S3 versioning, cross-region replication
• Multi-region support: Mumbai (ap-south-1) primary, Singapore (ap-southeast-1) disaster recovery
• SSL/TLS management: cert-manager with Let's Encrypt ClusterIssuer
• Secrets management: Kubernetes ExternalSecrets with AWS Secrets Manager
• Security: Falco runtime security, network policies, pod security standards, WAF rules

Out of Scope

• Application-level code changes (covered by service-specific epics)
• Third-party SaaS tool procurement (Datadog, New Relic alternatives)
• Physical hardware/on-premise deployment
• Cost optimization automation (reserved instances, spot fleet) -- future consideration

Current Implementation Status

• Terraform: 8 modules fully defined -- networking (VPC with public/private subnets, NAT gateway, security groups), compute (EKS cluster), database (RDS PostgreSQL with read replicas), cache (ElastiCache Redis cluster), storage (S3 buckets + lifecycle policies), CDN (CloudFront distribution), WAF (rate limiting + SQL injection rules), multi-region (secondary EKS, Route53 health-check failover, S3 cross-region replication). Three environment configurations: dev, staging, prod. Remote state in S3 with DynamoDB lock.
• Kubernetes: Base manifests with namespaces (hmp-dev, hmp-staging, hmp-prod), RBAC (cluster roles, service accounts), configmaps, secret templates, network policies, pod security policies. Addons: ArgoCD, cert-manager (ClusterIssuer), external-dns, Falco (custom rules), ingress-nginx, prometheus-stack.
• Helm: Generic hmp-service chart with templates for Deployment, Service, HPA, PDB, NetworkPolicy, ServiceMonitor, Argo Rollout, and analysis template. 14 per-service value files (api-gateway, auth, property, reservation, guest, frontdesk, housekeeping, POS, banquet, channel, CRM, notification, analytics, pgbouncer). Secondary region override values for ap-southeast-1.
• CI/CD: GitHub Actions workflows for terraform-plan, terraform-apply, and security-scan. ArgoCD Application manifests for dev, staging, and prod environments.
• Docker: docker-compose.demo.yml (11 containers: backend, dashboard, booking-engine, docs, demo-hub, PostgreSQL, Redis, Kafka, Zookeeper, Elasticsearch, Nginx). docker-compose.dev.yml for local development.
• Monitoring: Prometheus values with scrape configs and recording rules. Grafana dashboards (hmp-overview, hmp-business, hmp-sla). Jaeger values for distributed tracing. Loki values for log aggregation. OpenTelemetry Collector config. Alert rules (hmp-alerts, hmp-sla-alerts).
• Load Testing: k6 scripts with 8 scenario files covering all major service flows. Shared helpers for auth tokens, config, and data generation.
• Security: Falco custom rules for runtime anomaly detection. Network policies restricting inter-service communication. Pod security standards enforced.

Gap Analysis

• CI/CD End-to-End Validation: GitHub Actions workflows and ArgoCD manifests are configured, but the full pipeline (code push -> Docker build -> image push to ECR -> ArgoCD sync -> canary rollout -> promotion) has not been validated in a production-like environment. The Argo Rollout analysis template references Prometheus metrics that need real traffic to evaluate.
• Performance Baselines: k6 load testing scripts exist for 8 scenarios, but no baseline benchmarks have been established. Without baselines, there is no way to detect performance regressions or set meaningful SLO thresholds.
• Multi-Region DR Drill: Terraform multi-region module provisions secondary EKS in Singapore with Route53 failover, but no actual failover drill has been executed. The runbook directory exists but is empty.
• Per-Service Grafana Dashboards: Only 3 dashboards exist (overview, business, SLA). Individual service dashboards (e.g., reservation latency by endpoint, POS order throughput, channel sync success rates) are missing.
• Alerting Coverage: Alert rules exist for SLA violations but PagerDuty integration is not configured, and alert routing (which team gets paged for which service) is not defined.
• Database Migration Safety: Prisma migrations run via prisma migrate deploy but there are no safeguards for production (pre-migration backup, migration dry-run, rollback procedure, lock timeout configuration).
• Log Correlation: Loki and Jaeger are configured independently. Trace-to-log correlation (linking a Jaeger trace ID to Loki log entries) is not wired up via the OpenTelemetry Collector.
• Docker Demo Hardening: Demo stack works but uses plain-text secrets in compose file, no health check retry limits, and no resource limits on containers.

User Stories

Story ID	Title	Priority	Points	Status	Sprint
US-020-001	Docker Compose Demo Environment	P0	8	Built	1
US-020-002	Terraform IaC for Core Infrastructure	P0	13	Built	2
US-020-003	Kubernetes Base Manifests & Addons	P0	8	Built	3
US-020-004	Helm Chart per Microservice	P0	8	Built	3
US-020-005	CI/CD Pipeline (GitHub Actions + ArgoCD)	P0	8	Partial	4
US-020-006	Monitoring Stack (Prometheus + Grafana + Jaeger)	P0	8	Partial	5
US-020-007	Alerting & Incident Response (PagerDuty)	P1	5	Partial	6
US-020-008	Load Testing with Performance Baselines	P1	5	Partial	6
US-020-009	Multi-Region Disaster Recovery	P1	8	Partial	7
US-020-010	Database Migration Safety for Production	P1	5	Not Started	8
US-020-011	Log Correlation & Structured Observability	P2	5	Not Started	9

Dependencies

• EP-001 (Auth & Access): Service accounts and RBAC in Kubernetes depend on auth service configuration
• All backend epics: Every microservice is deployed via Helm charts and CI/CD pipelines defined here
• Cloud provider: AWS account with appropriate IAM permissions, billing approval for multi-region resources
• Third-party accounts: PagerDuty, container registry (ECR), DNS registrar

Acceptance Criteria (Epic Level)

• Demo environment spins up with docker compose up and all 11 containers pass health checks within 120 seconds
• terraform plan runs cleanly for dev, staging, and prod environments with no drift
• Kubernetes cluster bootstraps with all addons (ArgoCD, cert-manager, Falco, ingress-nginx, Prometheus) in under 15 minutes
• Code merged to main triggers automated build, image push, and canary rollout via ArgoCD
• Grafana dashboards show real-time metrics for every deployed microservice
• PagerDuty alerts fire within 2 minutes of an SLA breach (p99 latency > 500ms or error rate > 1%)
• k6 load tests establish baselines and run in CI to catch regressions exceeding 20% degradation
• Multi-region failover completes within 5 minutes (RPO < 1 hour, RTO < 5 minutes)
• Production database migrations include pre-backup, dry-run validation, and documented rollback

KPIs / Success Metrics

• Infrastructure uptime: 99.95% (SLA target)
• Deployment frequency: >= 5 deploys/week to production
• Deployment lead time (commit to production): < 30 minutes
• Mean time to recovery (MTTR): < 15 minutes for P0 incidents
• Failed deployment rate: < 5% (auto-rolled-back by Argo Rollout analysis)
• k6 p95 API response time: < 200ms under 500 concurrent users
• Monitoring coverage: 100% of services have Prometheus ServiceMonitor + Grafana dashboard
• Alert noise ratio: < 10% false positive alerts per week

---

Detailed User Stories

US-020-001: Docker Compose Demo Environment

As a developer or sales engineer, I want to spin up the entire HMP platform locally with a single docker compose up command, So that I can demonstrate the product, onboard new developers, and run integration tests without cloud infrastructure.

Priority: P0 | Points: 8 | Sprint: 1 | Status: Built

Implementation Status

Docker Compose demo stack (docker/docker-compose.demo.yml) orchestrates 11 containers: unified backend (all NestJS microservices started in entrypoint), Next.js dashboard, Vite booking engine, Docusaurus docs, static demo hub, PostgreSQL 15, Redis 7, Kafka + Zookeeper, Elasticsearch 8, and Nginx reverse proxy. Fixed tenant ID 00000000-0000-4000-a000-000000000001. Port mapping: Backend 3010, Dashboard 3020, Booking 3040, Docs 3050, Demo Hub 8000. Separate docker-compose.dev.yml for local development with hot-reload volumes.

Acceptance Criteria

Scenario: Full demo stack starts successfully
  Given the user has Docker Desktop running with at least 8 GB RAM allocated
  And the user is in the "docker/" directory
  When the user runs "docker compose -f docker-compose.demo.yml up -d"
  Then all 11 containers reach "healthy" status within 120 seconds
  And the demo hub is accessible at http://localhost:8000
  And the dashboard login page loads at http://localhost:3020

Scenario: Demo credentials authenticate successfully
  Given the demo stack is running and healthy
  When the user logs into the dashboard with "admin@hmphotel.com" / "admin123"
  Then the user sees the front desk dashboard for the demo property
  And the tenant ID header "x-tenant-id" is set to "00000000-0000-4000-a000-000000000001"

Scenario: Demo stack tears down cleanly
  Given the demo stack is running
  When the user runs "docker compose -f docker-compose.demo.yml down -v"
  Then all containers stop and all named volumes are removed
  And no orphan networks remain

---

US-020-002: Terraform IaC for Core Infrastructure

As a DevOps engineer, I want to provision all AWS infrastructure (VPC, EKS, RDS, ElastiCache, S3, CloudFront, WAF) using Terraform modules, So that infrastructure is version-controlled, reproducible, and auditable across dev/staging/prod environments.

Priority: P0 | Points: 13 | Sprint: 2 | Status: Built

Implementation Status

Eight Terraform modules under 06-hmp-infrastructure/terraform/modules/: networking (VPC with 3 AZs, public/private subnets, NAT gateway, security groups), compute (EKS with managed node groups), database (RDS PostgreSQL multi-AZ with read replicas), cache (ElastiCache Redis cluster mode), storage (S3 with lifecycle policies), CDN (CloudFront with S3 origin and custom domain), WAF (rate limiting, SQL injection, XSS rules), multi-region (secondary EKS, Route53 health-check failover, S3 cross-region replication). Three environment configs (dev, staging, prod) under terraform/environments/. Remote state stored in S3 with DynamoDB state locking via terraform/global/.

Acceptance Criteria

Scenario: Terraform plan for dev environment shows no errors
  Given the DevOps engineer has valid AWS credentials configured
  And the Terraform state backend (S3 + DynamoDB) has been bootstrapped
  When the engineer runs "terraform plan" in the dev environment directory
  Then the plan completes without errors
  And the plan shows the expected resource count for networking, compute, database, cache, and storage modules

Scenario: Staging environment provisions with encryption at rest
  Given the staging Terraform configuration specifies "encrypted = true" for RDS and S3
  When the engineer applies the staging configuration
  Then the RDS instance has storage encryption enabled with AWS-managed KMS key
  And all S3 buckets have default SSE-S3 encryption enabled
  And the ElastiCache cluster has at-rest encryption enabled

Scenario: Environment isolation via separate VPCs
  Given dev, staging, and prod each have their own Terraform workspace
  When all three environments are provisioned
  Then each environment runs in a separate VPC with non-overlapping CIDR ranges
  And there is no VPC peering between dev and prod

---

US-020-003: Kubernetes Base Manifests & Addons

As a platform engineer, I want to define Kubernetes base manifests (namespaces, RBAC, secrets, network policies, pod security) and cluster addons (ArgoCD, cert-manager, Falco, ingress-nginx, Prometheus), So that the cluster has a secure, observable foundation before any application workloads are deployed.

Priority: P0 | Points: 8 | Sprint: 3 | Status: Built

Implementation Status

Base manifests under 06-hmp-infrastructure/kubernetes/base/: namespaces (hmp-dev, hmp-staging, hmp-prod, hmp-monitoring, hmp-system), RBAC (cluster roles for admin/developer/readonly, service accounts per namespace), configmaps (shared app config), secret templates (database, Redis, auth secrets with ExternalSecrets CRDs), network policies (deny-all default with explicit allow rules per service pair), pod security (restricted PSS baseline enforced). Addons under kubernetes/addons/: ArgoCD, cert-manager (Let's Encrypt ClusterIssuer for staging and production), external-dns, Falco (custom rules for crypto mining detection, sensitive file access, privilege escalation), ingress-nginx (custom values with proxy protocol, rate limiting), prometheus-stack.

Acceptance Criteria

Scenario: Namespace isolation prevents cross-environment access
  Given the Kubernetes cluster has namespaces hmp-dev, hmp-staging, and hmp-prod
  When a pod in hmp-dev attempts to connect to a service in hmp-prod
  Then the connection is denied by the NetworkPolicy
  And a Falco alert is generated for the unexpected network connection

Scenario: ExternalSecrets sync from AWS Secrets Manager
  Given an ExternalSecret resource references "hmp/prod/database-url" in AWS Secrets Manager
  When the ExternalSecrets operator reconciles
  Then a Kubernetes Secret is created in the hmp-prod namespace with the database connection string
  And the secret is refreshed automatically every 15 minutes

Scenario: cert-manager provisions TLS certificates
  Given the ClusterIssuer "letsencrypt-prod" is configured with an ACME solver
  When an Ingress resource with annotation "cert-manager.io/cluster-issuer: letsencrypt-prod" is created
  Then cert-manager provisions a valid TLS certificate within 5 minutes
  And the certificate auto-renews 30 days before expiry

---

US-020-004: Helm Chart per Microservice

As a DevOps engineer, I want to deploy each NestJS microservice using a shared Helm chart with per-service value overrides, So that deployments are consistent, configurable, and support canary rollouts with automated analysis.

Priority: P0 | Points: 8 | Sprint: 3 | Status: Built

Implementation Status

Generic hmp-service Helm chart under 06-hmp-infrastructure/helm/hmp-service/ with templates: Deployment (or Argo Rollout for canary), Service (ClusterIP), HPA (CPU/memory-based autoscaling), PDB (minAvailable: 1), NetworkPolicy, ServiceAccount, ServiceMonitor (Prometheus scrape config), AnalysisTemplate (success rate + latency metrics), preview Service (for canary traffic). Per-service values for 14 services under helm/values/. Secondary region overrides at helm/values/ap-southeast-1/values.yaml.

Acceptance Criteria

Scenario: Helm template renders valid manifests for each service
  Given the hmp-service chart and the auth-service values file
  When the engineer runs "helm template auth ./hmp-service -f values/auth-service.yaml"
  Then the output contains a valid Deployment with the correct Docker image tag
  And the output contains an HPA with minReplicas and maxReplicas from the values file
  And the output contains a ServiceMonitor targeting port "metrics"

Scenario: Canary rollout with analysis gate
  Given the auth-service is deployed via Argo Rollout with canary strategy
  When a new image version is pushed
  Then ArgoCD syncs and creates a canary ReplicaSet with 20% traffic weight
  And the AnalysisTemplate evaluates success rate (> 99%) and p99 latency (< 500ms) over 5 minutes
  And if analysis passes, the rollout promotes to 100% automatically

Scenario: PDB prevents total service disruption during node drain
  Given the reservation-service has PDB minAvailable set to 1
  And the service has 3 replicas running across 2 nodes
  When a node is cordoned and drained
  Then at least 1 pod remains running and serving traffic throughout the drain

---

US-020-005: CI/CD Pipeline (GitHub Actions + ArgoCD)

As a developer, I want to merge code to main and have it automatically built, tested, scanned, and deployed to staging (then promoted to production via approval), So that releases are fast, safe, and require no manual intervention for standard deployments.

Priority: P0 | Points: 8 | Sprint: 4 | Status: Partial

Implementation Status

GitHub Actions workflows exist: terraform-plan.yml (runs on PR to validate Terraform changes), terraform-apply.yml (applies on merge to main), security-scan.yml (Trivy container scan, Checkov IaC scan). ArgoCD Application manifests for dev, staging, and prod. However, the application build/push workflow (compile NestJS -> Docker build -> push to ECR -> update Helm values -> ArgoCD sync) is not present as a unified pipeline. The ArgoCD deploy step is configured but the sync webhook from GitHub Actions to ArgoCD is not wired. Image tag promotion between environments is manual.

Acceptance Criteria

Scenario: PR triggers lint, test, and security scan
  Given a developer opens a pull request against the main branch
  When GitHub Actions workflows trigger
  Then the pipeline runs ESLint, unit tests, and Trivy security scan in parallel
  And the PR status checks block merge if any step fails
  And the Trivy scan reports any HIGH or CRITICAL CVEs as PR comments

Scenario: Merge to main triggers automated staging deployment
  Given the PR is approved and merged to main
  When the CI pipeline completes Docker image build and pushes to ECR with tag "main-<sha>"
  Then ArgoCD detects the image tag update in the staging Application manifest
  And a canary rollout begins in the hmp-staging namespace
  And the deployment completes within 10 minutes with zero downtime

Scenario: Production promotion requires manual approval
  Given the staging deployment has been running for at least 1 hour with no alert firing
  When the release manager approves the production promotion in ArgoCD
  Then the production Application manifest is updated with the same image tag
  And the canary rollout runs in hmp-prod with the AnalysisTemplate gate

---

US-020-006: Monitoring Stack (Prometheus + Grafana + Jaeger)

As an SRE engineer, I want to monitor all microservices with metrics, dashboards, and distributed traces, So that I can detect performance degradation, debug latency issues, and maintain SLA compliance.

Priority: P0 | Points: 8 | Sprint: 5 | Status: Partial

Implementation Status

Prometheus is configured with scrape configs for all ServiceMonitors and recording rules for key aggregations. Three Grafana dashboards exist: hmp-overview (cluster-wide health), hmp-business (occupancy, revenue, bookings), hmp-sla (p99 latency, error rate, availability by service). Jaeger is deployed for distributed tracing. Loki is configured for log aggregation. OpenTelemetry Collector values exist. Gap: per-service dashboards (reservation latency breakdown by endpoint, POS order throughput, channel sync success/failure) are not built. Trace-to-log correlation via OTel is not wired.

Acceptance Criteria

Scenario: Every microservice exposes Prometheus metrics
  Given all 14 microservices are deployed with the hmp-service Helm chart
  When Prometheus scrapes the /metrics endpoint on each service
  Then metrics for http_request_duration_seconds, http_requests_total, and nodejs_heap_size_bytes are available
  And Grafana overview dashboard shows all services as "UP"

Scenario: Per-service Grafana dashboard shows endpoint-level metrics
  Given the reservation-service Grafana dashboard is deployed
  When a user navigates to the dashboard and selects the reservation-service
  Then the dashboard displays p50/p95/p99 latency broken down by endpoint (GET /reservations, POST /reservations, etc.)
  And error rate by status code (4xx, 5xx) is visible
  And the dashboard supports time range selection and auto-refresh

Scenario: Distributed trace spans across API Gateway to downstream service
  Given a guest makes a booking via the booking engine
  When the request passes through api-gateway -> reservation-service -> guest-service
  Then a single Jaeger trace contains spans for all three services
  And each span includes the tenant ID, request ID, and duration

---

US-020-007: Alerting & Incident Response (PagerDuty)

As an on-call engineer, I want to receive PagerDuty alerts when SLA thresholds are breached or services go down, So that I can respond to incidents within the target MTTR of 15 minutes.

Priority: P1 | Points: 5 | Sprint: 6 | Status: Partial

Implementation Status

Prometheus alert rules defined in monitoring/alerts/hmp-alerts.yaml and hmp-sla-alerts.yaml covering: service down (probe failure > 1 min), high error rate (> 1% 5xx for 5 min), high latency (p99 > 500ms for 5 min), pod crash loop, disk pressure, certificate expiry (< 14 days). Alert rules fire to Alertmanager. Gap: Alertmanager is not configured with PagerDuty receiver. No routing rules to map service alerts to team escalation policies. No runbooks linked from alerts.

Acceptance Criteria

Scenario: High error rate triggers PagerDuty incident
  Given the reservation-service error rate exceeds 1% for 5 consecutive minutes
  When Prometheus evaluates the alert rule "HMPHighErrorRate"
  Then Alertmanager routes the alert to the PagerDuty "backend-team" service
  And a PagerDuty incident is created with severity "critical"
  And the incident description includes the service name, current error rate, and link to Grafana dashboard

Scenario: Alert auto-resolves when condition clears
  Given a PagerDuty incident was created for "HMPHighLatency" on auth-service
  When the p99 latency drops below 500ms for 5 minutes
  Then Alertmanager sends a "resolved" notification to PagerDuty
  And the incident is auto-resolved in PagerDuty

Scenario: Certificate expiry alert fires 14 days before expiry
  Given a TLS certificate managed by cert-manager has 13 days until expiry
  When Prometheus evaluates the "CertificateExpiringSoon" alert
  Then a warning-severity alert is sent to the platform-team PagerDuty service
  And the alert includes the certificate name, namespace, and expiry date

---

US-020-008: Load Testing with Performance Baselines

As a performance engineer, I want to run k6 load tests against all major service flows and establish baseline performance benchmarks, So that I can detect regressions in CI and capacity-plan for peak season (Diwali, New Year).

Priority: P1 | Points: 5 | Sprint: 6 | Status: Partial

Implementation Status

k6 load testing scripts under 06-hmp-infrastructure/load-testing/scenarios/ covering 8 flows: auth (login, token refresh), reservations (search, create, modify), booking-engine (availability check, booking flow), frontdesk (check-in, check-out), housekeeping (task assignment, status update), POS (order create, payment), channel (rate push, availability sync), full-load (combined scenario). Shared helpers for auth token generation, config (target URLs, VU counts), and test data generation. Gap: no baseline results recorded, no CI integration, no threshold definitions in k6 scripts, no comparison reports.

Acceptance Criteria

Scenario: Baseline benchmark is recorded for reservation flow
  Given the k6 reservation scenario runs against the staging environment
  When the test executes with 100 virtual users for 10 minutes
  Then the results are saved to a JSON report including p50, p95, p99 latency, throughput (req/s), and error rate
  And the baseline is stored in the repository as "baselines/reservations.json"

Scenario: CI pipeline fails on performance regression
  Given a baseline exists for the auth flow with p95 < 150ms
  When a code change causes auth p95 latency to increase to 250ms (67% regression)
  Then the k6 CI step compares against the baseline
  And the pipeline fails with a clear message: "Auth p95 latency regression: 150ms -> 250ms (67% increase, threshold: 20%)"

Scenario: Full-load test simulates peak season traffic
  Given the full-load scenario combines all 8 flows with weighted distribution
  When the test ramps to 500 concurrent virtual users over 5 minutes and sustains for 15 minutes
  Then the system maintains p99 latency < 500ms and error rate < 0.1%
  And resource utilization (CPU, memory) per service is recorded for capacity planning

---

US-020-009: Multi-Region Disaster Recovery

As a platform engineer, I want to validate the multi-region failover from Mumbai (ap-south-1) to Singapore (ap-southeast-1), So that the platform can survive a regional outage with RPO < 1 hour and RTO < 5 minutes.

Priority: P1 | Points: 8 | Sprint: 7 | Status: Partial

Implementation Status

Terraform multi-region module provisions: secondary EKS cluster in ap-southeast-1, Route53 health-check-based DNS failover, S3 cross-region replication for static assets, RDS cross-region read replica (configured in database module). Helm values override for Singapore region at helm/values/ap-southeast-1/values.yaml. Gap: no DR drill has been executed. Runbook directory (monitoring/runbooks/) exists but is empty. RDS promotion from read replica to primary is not automated. Kafka (MSK) cross-region replication is not configured. Redis cache warm-up strategy after failover is undefined.

Acceptance Criteria

Scenario: DNS failover routes traffic to Singapore when Mumbai is unreachable
  Given the Route53 health check monitors the Mumbai API Gateway endpoint
  When the Mumbai endpoint fails health checks for 3 consecutive intervals (30 seconds each)
  Then Route53 updates the DNS record to point to the Singapore API Gateway
  And new client requests resolve to the Singapore region within 60 seconds (TTL)

Scenario: RDS read replica promotion in Singapore
  Given the Singapore region has an RDS read replica replicating from Mumbai
  When the DR runbook is executed (manual or automated trigger)
  Then the read replica is promoted to a standalone primary within 5 minutes
  And the application configuration is updated to use the new primary endpoint
  And data loss is limited to the replication lag (target: < 1 minute under normal load)

Scenario: DR drill completes successfully with documented results
  Given the DR drill is scheduled for a maintenance window
  When the team executes the full failover procedure (DNS switch, RDS promotion, service verification)
  Then all API endpoints respond successfully from the Singapore region
  And the drill results are documented with actual RPO, RTO, and any issues encountered
  And the runbook is updated based on lessons learned

---

US-020-010: Database Migration Safety for Production

As a backend developer, I want to run Prisma database migrations in production with automated safeguards (pre-backup, dry-run, rollback plan), So that schema changes do not cause data loss or extended downtime.

Priority: P1 | Points: 5 | Sprint: 8 | Status: Not Started

Implementation Status

Currently, prisma migrate deploy is run manually or in the container entrypoint without safeguards. No pre-migration backup trigger. No dry-run or shadow database validation against production schema. No lock timeout configuration (long-running migrations can lock tables and block queries). No automated rollback procedure. The Prisma schema has 40+ models; some migrations involve adding columns with defaults, which on large tables (e.g., reservations, guests, folio_entries) can be slow.

Acceptance Criteria

Scenario: Pre-migration backup is created automatically
  Given a new Prisma migration is pending for the production database
  When the migration pipeline initiates
  Then an RDS snapshot is created and tagged with the migration name and timestamp
  And the pipeline waits for the snapshot to reach "available" status before proceeding
  And the snapshot is retained for 30 days

Scenario: Migration dry-run validates against production schema
  Given the migration adds a NOT NULL column to the "reservations" table (500K+ rows)
  When the dry-run step executes against a schema-only clone of production
  Then the dry-run reports estimated execution time and lock duration
  And if estimated lock time exceeds 30 seconds, the pipeline pauses and notifies the team
  And the team can approve, modify (e.g., add column as nullable first), or abort

Scenario: Failed migration triggers automatic rollback
  Given a migration is running and encounters an error (e.g., constraint violation)
  When the migration fails midway
  Then the pipeline restores the database from the pre-migration snapshot
  And a PagerDuty incident is created with the migration error details
  And the failed migration is marked as "rolled back" in the deployment log

---

US-020-011: Log Correlation & Structured Observability

As an SRE engineer, I want to correlate logs, traces, and metrics using a shared request ID and trace ID, So that I can jump from a Grafana metric spike to the Jaeger trace to the exact Loki log lines in one click.

Priority: P2 | Points: 5 | Sprint: 9 | Status: Not Started

Implementation Status

Each component exists independently: Prometheus collects metrics, Jaeger collects traces, Loki collects logs. The OpenTelemetry Collector config is defined but not fully wired. NestJS services do not inject trace context (traceId, spanId) into structured log output. Grafana data source linking (Loki -> Jaeger derived fields, metric -> trace exemplars) is not configured.

Acceptance Criteria

Scenario: Log entries include trace ID for cross-referencing
  Given the api-gateway receives a request with OpenTelemetry trace context
  When the request is processed and logged
  Then the JSON log entry includes fields "traceId", "spanId", "requestId", and "tenantId"
  And the same traceId appears in logs from all downstream services that handled the request

Scenario: Grafana Loki links to Jaeger trace
  Given a log entry in Loki contains a traceId field
  When the engineer clicks the traceId value in the Grafana Loki explore view
  Then Grafana opens the corresponding Jaeger trace in a split view
  And the trace shows all spans across api-gateway, reservation-service, and guest-service

Scenario: Prometheus exemplars link metrics to traces
  Given Prometheus collects http_request_duration_seconds with exemplar support enabled
  When the engineer views a latency spike on the Grafana dashboard
  Then the spike data points include exemplar trace IDs
  And clicking an exemplar opens the corresponding Jaeger trace