Skip to main content

MASTER PENDING TRACKER — 2026-05-20

Authored: 2026-05-20. Supersedes: all prior 2026-05-18 / 2026-05-20 checklists (preserved for history). Purpose: Single source of truth for every open work item. Tracks until 100% closure for first-property launch. Mutating this file is OK — flip status as items close, add new rows as new items surface. Don't re-snapshot; this IS the rolling tracker.

Per-item handoff (for new sessions picking up a single item)​

For the 5 hard blockers in §1, single-file handoffs exist so a new session can execute without re-discovering context:

ItemHandoff fileType
FU-46 RS256 rotationhmp-infrastructure/monitoring/runbooks/fu-46-rs256-rotation-playbook.mdEngineering playbook (step-by-step + scripts)
FU-51 SendGrid prod keyhmp-infrastructure/monitoring/runbooks/fu-51-sendgrid-prod-key-playbook.mdEngineering playbook
FU-53 metrics-server test+qahmp-infrastructure/monitoring/runbooks/fu-53-metrics-server-rollout-playbook.mdEngineering playbook (mirror FU-52)
Q4 on-call rotationAuto-loaded via memory/project_q4-on-call-rotation-handoff.mdStakeholder decision (auto-loaded in MEMORY.md)
Q5 backup retentionAuto-loaded via memory/project_q5-backup-retention-handoff.mdStakeholder decision (auto-loaded in MEMORY.md)

A new session asking "let's work on FU-46" should read the playbook FIRST — ~5 min context vs ~20 min without.

For other items (P0/P1/P2/L1-L4/FUs), the master tracker rows below give title + impact + acceptance. Linked files in each row provide the full procedure.

How to read this​

Status:

  • ✅ DONE — shipped + verified in code
  • 🟢 IN PROGRESS — work actively underway
  • 🟡 OPEN — authored / planned, needs operator or stakeholder action
  • 🔴 NOT STARTED — nothing done yet
  • ❓ VERIFY — status uncertain; needs code/state verification
  • 🛑 BLOCKED — waiting on external dependency (vendor, stakeholder, procurement)
  • ⛔ OPS-GATED — needs operator-side execution (SSH, prod cred, manual click)

Priority: P0 = launch-blocking · P1 = must-fix-soon · P2 = nice-to-have · FU = follow-up · L1-L4 = engineering hardening

Effort: S = ≤1 day · M = 1-3 days · L = 1 week+ · XL = >2 weeks

Weight (calc): priority × impact / effort. Larger = better ROI. Used to rank when planning a session.

  • Priority value: P0=10, P1=5, P2=2, FU=4, L1=8, L2=6, L3=5, L4=3
  • Impact value: launch-blocking=10, post-launch=5, polish=2
  • Effort value: S=1, M=3, L=8, XL=20

Owner abbreviations: Op=Operator/SRE · Eng=Engineering · CFO/CISO/VP/Legal/CMO etc = stakeholders · Vendor=external

TABLE OF CONTENTS​

  1. Active hard blockers (date-bound)
  2. Property #1 audit — P0 (47 items, individual status)
  3. Property #1 audit — P1 (41 items)
  4. Property #1 audit — P2 (39 items)
  5. Plan 4/5/6/7 status
  6. Follow-ups (FU-2 .. FU-53)
  7. Engineering hardening for scale (L1–L4)
  8. Stakeholder sign-offs (Q1–Q5, CIC-1..5, Sales+Marketing)
  9. Coverage gates (Property #1 gold standard charter)
  10. Top 25 prioritized actions (weighted ranking)
  11. Verification needed (status uncertain)
  12. Prod-onboarding fixes — LOTUS LEAF RESORT (2026-05-22)
  13. Property #1 Dashboard Readiness — PA/PB/PC/PD (2026-05-23 → 24)
  14. AP-ARCH — architecture remediation track (30 items, 2026-06-09, ADDITIVE)

1. Active hard blockers​

Date-bound items that MUST close before first-property onboarding (target ~2026-06-15).

IDItemDeadlineOwnerStatusImpact
FU-51Provision real SendGrid prod live key via AWS SSM keynova/prod/sendgrid + apply hmp-third-party-secrets ExternalSecret. Stub key SG.stub-not-real-pending-fu-51 still in kubernetes/overlays/prod/hmp-notification-service/kustomization.yaml.2026-06-15Op🟡 OPENReal outbound email silently 401s — guest confirmations, IRN delivery, password resets all broken
FU-46RS256 JWT keys + remove ALLOW_HS256 stopgap.2026-06-15Op + Eng✅ DONE 2026-05-21RS256 keypairs provisioned in ALL 4 envs via scripts/fu-46-provision-env.sh (prod done manually during the bootstrap marathon) — auth-service holds private+public, all other backends hold public. Folded into Terraform (tls_private_key.jwt in each env's service-runtime-secrets.tf, commit 5197318) so from-scratch rebuilds get RS256 natively. ALLOW_HS256: "true" removed from all 16 overlay configmaps (995ec4a). Playbook rewritten to point at the script + warn against the awk-PEM bug (2ed6407). staging+qa dashboard login Playwright-verified on RS256.
Reloader prod validationForce-rotate test cluster RDS to validate Reloader works end-to-end + cover all 20 prod backends.2026-05-28 (actual prod NextRotation per AWS Secrets Manager; the earlier "2026-05-24" guess was stale)Op✅ DONE 2026-05-23(1) End-to-end chain validated on keynova-test 12:56 IST: rotate-secret --rotate-immediately → RDS new password (OwningService=rds, no Lambda) → ExternalSecrets re-synced within ~30s → K8s Secret resourceVersion 5510370→10048760 → Reloader v1.4.5 (not v2.1.5; v1.x is the actual modern line) detected at 07:27:26Z and triggered rolling restarts for 17 RDS-touching deployments within 1.5s (3 frontend apps correctly skipped) → all pods came back 1/1 Ready → API smoke 200 throughout. (2) Prod pre-flight discovered 6 of 20 prod backends MISSING the reloader.stakater.com/auto annotation (hmp-analytics-service, hmp-banquet-service, hmp-channel-service, hmp-crm-service, hmp-revenue-strategy, hmp-spa-service) — annotation was in git base since e66b694 (FU-52 2026-05-17) but ArgoCD's per-overlay change-detection had missed it for these 6. Without the fix, 6 of 20 deployments would have been silently skipped on 2026-05-28 rotation → CrashLoopBackOff on next restart. (3) Fixed immediate via kubectl annotate × 6 + durable via touch-overlays commit hmp-infrastructure 37101b3 (force ArgoCD re-render). All 20 deployments now verified annotated post-ArgoCD reconcile, no rolling restarts triggered (live state matched git). (4) ExternalSecrets refresh interval on prod RDS secret tightened from 5m → 1m to shrink rotation lag window. Setup side-effects: keynova-test cluster auth mode flipped CONFIG_MAP → API_AND_CONFIG_MAP; break-glass SSO role got AmazonEKSClusterAdminPolicy Access Entry on test cluster (one-time, persistent). New durable feedback memory: [[feedback_argocd-misses-base-changes-without-overlay-touch]].
Plan 6 Q4 (on-call rotation)Set up rotation with ≥3 named approvers (2-of-N gate). Currently 1 named human.2026-06-15VP Eng + SRE Lead🛑 BLOCKEDPlan 4 prod-demo + Plan 7 destroys + AP-T17-* all blocked behind 2-of-N approvals
Plan 6 Q5 (backup retention)Legal/DPO sign-off on current 30d prod / 7y CloudTrail retention recommendation2026-06-15Legal/DPO + CISO + SRE Lead🛑 BLOCKEDDPDPA/SOC 2 attestation pending
Plan 5 7-day burn-inResAvenue cert v3 + live.keynova.ai 7-day burn-in monitoring2026-05-24 (day 3 of 7 today)Op🟠 PARTIAL (reconciled 2026-06-04)Closes FU-6; clears Plan 7 cleanup

2. Property #1 audit — P0 (47 items)​

Status reflects: 2026-05-08 deferred doc + git log AP-T closures across 18 repos (374 commits) + 2026-05-09 coverage doc + 2026-05-20 cluster state. Many items shipped post-audit; many remain.

ID | Title | Affected flows | Effort | Status | Impact

IDTitleFlowsEffortStatusImpact / Notes
AP-T01-001Make tenantId optional on ResetPasswordRequestDto; resolve user by email1S✅ DONEForgot-password works
AP-T01-002Subscribe notification dispatcher to TENANT_CREATED; render welcome + verify-email templates1M✅ DONEWelcome emails fire (audit/AP-T01-002)
AP-T01-003Allow ADMIN (not just SUPER_ADMIN) on OnboardingController1S✅ DONETenant admin can self-onboard
AP-T01-004Add (auth)/forgot-password/page.tsx + /signup in admin-portal1M✅ DONEVerified 2026-05-23 — both (auth)/signup/page.tsx + (auth)/forgot-password/page.tsx exist + wired in admin-portal
AP-T02-001Add latitude, longitude, googleMapsUrl to Property DTOs + schema2S✅ DONEVerified 2026-05-23 — all 3 fields in schema.prisma:115-117 + DTOs property.dto.ts:227-243
AP-T02-002Provision S3 (AWS) + MinIO sidecar local; add AWS_S3_* env; AssetType enum2M✅ DONEMinIO sidecar + c2a48d4 feat(infrastructure): wire AWS_S3_* env vars on hmp-asset-service
AP-T02-003Dashboard image-upload widget; replace URL-paste2M✅ DONE (reconciled 2026-06-04)Widget shipped; full 7-step wizard expansion still deferred (Google Maps key procurement-gated)
AP-T02-004Re-seed QA Property 40b29005-...2,4,10,11,12S✅ DONE7e0b9d5 seeds: comprehensive maximalist seed data + Plan 4 demo enrichment 2026-05-18..20
AP-T03-001Add idempotencyKey to Reservation; enforce on booking-engine3S✅ DONEVerified 2026-05-23 — idempotencyKey schema.prisma:125-126 + @@unique([propertyId, idempotencyKey]):159
AP-T03-002Wire DB-backed IApiKeyService; rotate the shared seed key3M✅ DONEClosed end-to-end by A5 (2026-05-23, §12) — per-(tenant,property) publishable ApiKey with lazy get-or-create, delivered to the multi-tenant booking-engine SPA via resolve-host. No shared seed-key path remains; verified live on prod.
AP-T04-001Re-run seed-master.sh against QA RDS for Property #14S✅ DONEAuto-seed pipeline running on every change (2026-05-19)
AP-T04-002Decide: does Phase 1 require mobile reservations? Scaffold or descope4,6,9XL/S✅ DONE (reconciled 2026-06-04)Phase 1 mobile scope decision doc exists (phase1-mobile-scope-decision-2026-04-28.md); confirm explicit deferral signed
AP-T05-001Fix rate recalc in reservations.service.update() for extend stay5S✅ DONEVerified 2026-05-20 — reservations.service.ts:1690-1736 recomputes ratePerNight + totalAmount + taxAmount + balanceDue; 3 specs at reservations.service.spec.ts:917-987 pass
AP-T05-002Post EXTENSION_NIGHT folio CHARGE for added nights when CHECKED_IN5M✅ DONEVerified 2026-05-20 — reservations.service.ts:1805-1858 publishes folio.charge.posted with transactionCode='EXTENSION_NIGHT' per added night; consumer at pos-folio-charge.consumer.ts dedups via externalOrderId
AP-T05-003Nginx /ws/housekeeping/ + /ws/kitchen/ proxy_pass on QA + prod5,7S✅ DONE95b9570 audit(AP-T05-003): add /ws/housekeeping/ + /ws/kitchen/ proxy_pass routes
AP-T06-001FRRO portal SOP runbook + IndianFrroClient interface spec6S✅ DONEVerified 2026-05-24: runbook at hmp-docs/docs/operations/frro-form-c.md (128 lines); IndianFrroClient interface + ManualUploadFrroClient impl in @hmp/integrations/src/lib/frro/; wired into hmp-frontdesk-service/src/cform/. Commits 56223a8 + b55fb8d (2026-04-29).
AP-T06-002Replace idType=OTHER visa hack with dedicated GuestVisa model6M✅ DONE2026-05-23 — typed GuestVisa model + FRRO/C-Form fields + migration 20260523000000_guest_visa_frro_fields; branch audit/AP-T06-002-guest-visa-model (06eb705). Check-in-flow wiring deferred (needs guest-service merge + GuestServiceClient methods)
AP-T06-003Connect kiosk useCheckInFlow to real backend OR descope kiosk6L/S✅ DONE (reconciled 2026-06-04)Kiosk is mock; needs explicit "Phase 1 = staff check-in only" sign-off
AP-T06-004Mobile staff frontdesk screen OR descope mobile-staff for Property #16L/S✅ DONE (reconciled 2026-06-04)Same pattern — descope likely; needs sign-off
AP-T07-001Rewrite KDS Electron app: /kitchen namespace, JWT auth.token, real event names7M✅ DONEVerified 2026-05-20 — hmp-kds-kiosk/packages/kds/src/renderer/App.tsx:11 uses /kitchen namespace; useAuthToken() reads JWT from localStorage; useSocketEvent(socket, 'kot:created', ...) listens for real events. Component tests pass.
AP-T07-002Wire kitchenGateway.emitKotCreated() into OrdersService.generateKOT7S✅ DONEVerified 2026-05-20 — orders.service.ts:623 invokes this.kitchenGateway.emitKotCreated(...) in try/catch; also publishes POS_KOT_CREATED to Kafka. 50/50 specs pass.
AP-T07-003Drop /pos/ prefix from mobile F&B paths7S✅ DONEaudit/AP-T07-003-menu-item-hsn-gst-dto (7e367f0) + UI
AP-T08-001Replace irn = 'Pending' literal with real IRN + QR in PDF8S✅ DONEVerified 2026-05-20 — invoice-pdf.service.ts:540-579 reads folio.irn (AP-T08-006 denorm). When null renders "DRAFT — IRN PENDING" notice; when set, renders 64-char IRN + ackNumber + signedQRCode. 10/10 specs pass (stale auto-invoice-number test rewritten in same session).
AP-T08-002NIC creds env vars in staging-aws + prod-aws compose; EINVOICE_MOCK env-driven8S🟡 PARTIALEINVOICE_MOCK=false flipped (6681da7); NIC creds still missing per CLAUDE.md — operator must populate
AP-T08-003Wire GuestServiceClient.getCompany() for B2B IRN (replace null company)8M✅ DONEVerified 2026-05-20 — einvoice.service.ts:59-187 resolves company via Prisma cross-service include { company: true }, computes buyerGstin / B2B-vs-URP / inter-vs-intra-state IGST. 32/32 specs pass. STRICT-TODO at line 43 notes future migration to ServiceClient pattern but functional equivalence confirmed.
AP-T08-004GuestRetentionTimer row at checkOutDate+90d; cron erases8M✅ DONEVerified 2026-05-20 — checkout.service.ts:541 + guest-portal.service.ts:286 publish RESERVATION_CHECKED_OUT; gdpr/checkout-consumer.service.ts:21-74 creates timer; gdpr/retention-cron.service.ts:75 (@Cron 02:10 daily) erases PII at expiry; schema model GuestRetentionTimer at schema.prisma:692-704. 8/8 specs pass.
AP-T08-005Add GuestErasedConsumer to frontdesk, pos, housekeeping, analytics, crm, payment, notification (7 services)8M✅ DONE2026-05-23 — frontdesk GuestErasedConsumer (eca3feb) + notification (5f39f4b, also fixed a broken shipped spec). All 7 services now covered
AP-T09-001Extend NA rollback transactionCode filter to include OVERSTAY_NIGHT + NO_SHOW9S✅ DONEd0eca0b audit(AP-T09-001): runbook references actual NightAuditLog @@unique migration filename
AP-T09-002Wire analytics dispatcher to night.audit.completed; backfill MV9,10M✅ DONEVerified 2026-05-20 — night-audit.consumer.ts:52-233 refreshes MvDailyOccupancy + MvDailyKpi (the renamed MvAdrRevpar) + NightAuditLog on every event. Publisher at night-audit.service.ts:310-311 (outbox) + :360-361 (inline). 14/14 specs pass. Dispatcher subscribes 38 topics total.
AP-T10-001Wire MvAdrRevpar.adr/revpar on every upsert; backfill historical10S✅ DONE (reconciled 2026-06-04)Report card tile shipped (AP-T10-001-report-card-tile-reportids); verify MV columns populated
AP-T10-002DashboardController.getMetrics queries MvAdrRevpar (single source of truth)10S✅ DONEVerified 2026-05-20 — dashboard.service.ts reads MvAdrRevpar with live-folio day-zero fallback. 2/2 specs pass. CLAUDE.md Gotcha confirms 2026-04-28 shipping.
AP-T10-003Investigate QA empty-MV root cause10M✅ RESOLVED 2026-05-20Operator-executed the diagnostic runbook. Reality: 9 of 31 MVs ARE populated on QA (mv_revenue_by_source=140, mv_channel_performance=107, mv_daily_occupancy=34, mv_housekeeping_kpi=34, mv_commission=10, mv_daily_arrivals=4, mv_daily_kpi=2, mv_daily_bookings=2, mv_front_office_pace=2). FolioConsumer + ReservationConsumer + ChannelConsumer + HousekeepingConsumer + NightAuditConsumer are running end-to-end against 576 reservations + 491 folios + 3,310 folio_transactions. EventStore being empty is BY DESIGN — actual event flow is Redis Streams → consumers → MVs directly; event_store table is a residual schema. The original "/api/reports/* returns {data:[]}" symptom was either stale or about specific reports backed by the 22 empty MVs. Those split into 3 NEW follow-ups: FU-54 (POS/payment producer demo data), FU-55 (Plan-2 producer-side field emission), and lazy-cron MVs that populate on demand. NO analytics-service engineering fix needed.
AP-T10-004Fix mobile dashboardApi.getMetrics URL to /analytics/dashboard/metrics10S✅ DONEVerified 2026-05-20 — hmp-mobile-apps/packages/api-client/src/dashboardApi.ts:19 calls /analytics/dashboard/metrics; X-Property-ID header injected by axios interceptor from SecureStore. 2/2 specs pass. Shipped 2026-04-28.
AP-T10-005Build GSTR-3B prep endpoint behind /gst prefix10M✅ DONEVerified 2026-05-20 — hmp-analytics-service/src/gstr/ ships GstrController + GstrService + GstrModule. GET /reports/gstr3b?month=YYYY-MM&propertyId=UUID aggregates EventStore → MvGstr3bSnapshot cache. 7/7 specs pass. Shipped 2026-04-29.
AP-T11-001Add @All('intermediary'), @All('channel-stats'), @All('channel-setup'), @All('webhook-management') decorators in gateway11S✅ DONEVerified 2026-05-23 — all 4 @All decorators + services.config.ts entries present
AP-T11-002Provision real ResAvenue connection row for Property #1; set CHANNEL_MODE11M✅ DONEed88cfe audit(AP-T11-002): add ResAvenue simulator connection seed + PROD cert 2026-05-16
AP-T11-003Register outbound webhook URL with ResAvenue11S✅ DONE337b1a0 audit(AP-T11-003): configure webhook env vars in compose + update seed SQL
AP-T12-001Seed MerchantAccount for tenant 1 on QA with Razorpay sandbox creds12S✅ DONE1f6db25 audit(AP-T12-001): add Razorpay sandbox MerchantAccount API seed
AP-T12-002Admin onboarding wizard → encrypted MerchantAccount per tenant12M🛑 BLOCKEDProcurement: Finance must get Razorpay LIVE creds
AP-T13-001Tenant-scope assert on CalendarService.getCalendar + createQuickReservation13S✅ DONE97f5492 audit(AP-T13-001): unblock tape-chart Playwright suite — frontdesk boot guard + seed roomId patch
AP-T13-002Wire FilterChipsRow → query params → backend filter OR delete cosmetic13M✅ DONE0cb6adb audit(AP-T13-002): Bar.onClick + Ctrl+Click multi-select
AP-T13-003Tape-chart RoomBlocks "Block range" frontend13M✅ DONE6651ca9 audit(AP-T13-003): Full Block Range dialog + bulk room picker followup f9119c6
AP-CC-1CI check: gateway prefixes (services.config.ts ↔ @All decorator parity)allS✅ DONEVerified 2026-05-20 — scripts/validate-gateway-routing.ts ships full parity check; wired into .github/workflows/ci.yml:55 (npm run validate:gateway); 17/17 unit-spec tests pass + 18/18 routing tests pass. Same session fixed 2 P0 silent-404 drifts (distribution/allocations + dashboard/portfolio) caught by the live check, both now have explicit @All decorators.
AP-CC-2Reseed QA RDS via seed-master.sh; verify all data populated2,4,10,11,12S✅ DONEAuto-seed pipeline handles this; smoke test passes
AP-CC-3RS256 migration for all WebSocket gateways before prod5,7,13M✅ DONE 2026-05-21Closed with FU-46 — RS256 keypairs in all 4 envs; api-gateway/HK/POS WebSocket gateways hold JWT_PUBLIC_KEY; ALLOW_HS256 stopgap removed from all 16 overlays.
AP-CC-4Remove EINVOICE_MOCK=true + HARDWARE_MOCK=true from prod-aws; add NIC + Razorpay + ResAvenue + SendGrid placeholders7,8,11,12S✅ DONE6681da7 audit(AP-CC-4): flip EINVOICE_MOCK to false + 386f7a1 + 7a5891f
AP-CC-5Provision SendGrid creds across QA + prod; e2e test for invoice email1,8S🟡 PARTIALBoot-guard shipped (b05cbba audit(AP-CC-5)); stub key in prod overlay → same as FU-51
AP-CC-6SES production-access remediation — SNS bounce/complaint pipeline + forced From=no-reply@mail.keynova.ai + public AUP/Anti-Spam/Unsubscribe pages on keynova.ai + AWS Support resubmission. Spec hmp-docs/docs/superpowers/specs/2026-05-26-ap-cc-6-ses-prod-access-design.md, plan …/plans/2026-05-26-ap-cc-6-ses-prod-access-plan.md, use-case write-up …/platform-registry/ap-cc-6-ses-prod-access-use-case-2026-05-26.md (status=submitted-2026-05-31).HighEng🛑 BLOCKED 2026-05-31 — case resubmitted to AWS Support 2026-05-31; awaiting re-review (~24-48h). Code + infra + pages all live in prod.

P0 closure summary (updated 2026-05-20 — top-10 session):

  • ✅ DONE: ~35 items (+13 this session: AP-T05-001/002, AP-T07-001/002, AP-T08-001/003/004, AP-T09-002, AP-T10-002/003/004/005, AP-CC-1 — verified shipped + tests pass, or freshly fixed; see per-row notes + change-log)
  • 🟡 PARTIAL: AP-T02-003, AP-T08-002, AP-CC-5 — AP-T08-005 closed 2026-05-23 (frontdesk + notification consumers)
  • 🔴 NOT STARTED: 0 P0 items remain — AP-T06-002 closed 2026-05-23 (typed GuestVisa model)
  • 🛑 BLOCKED: 1 (AP-T12-002 procurement)
  • 🟡 DESCOPE PENDING: 3 (AP-T04-002, AP-T06-003, AP-T06-004 mobile/kiosk)
  • ❓ VERIFY: 0 — all cleared 2026-05-23 (AP-T01-004, AP-T02-001, AP-T03-001, AP-T10-001, AP-T11-001 verified ✅ DONE; AP-T03-002 closed by A5)

3. Property #1 audit — P1 (41 items)​

IDTitleEffortStatus
AP-T01-005Public GET /billing/plansS✅ DONE (cluster fully closed via 4-pass on 2026-05-08)
AP-T01-006(Other Flow 1 P1 items, see audit doc)—✅ DONE / REDUNDANT — verified 2026-05-24: placeholder row; all referenced Flow-1 items (T01-005, T01-007) are ✅. 2026-04-29 HTML index conflated with AP-T05-009 + AP-T12-007 — pure doc drift.
AP-T01-007OnboardingService xPrisma deprecation — delete or migrateM✅ DONE (verified 2026-05-23 already-shipped — migrated to PropertyServiceClient cd44244, xPrisma fully removed, 15/15 specs)
AP-T02-005Add inclusive Boolean column to TaxConfigurationS✅ DONE (1cec19a audit(AP-T02-005))
AP-T02-006Class-validator schema for CancellationPolicy.rules[]S✅ DONE (3269053 audit(AP-T02-006))
AP-T03-003Decide theme strategy: KeyNova brand OR document direction-a/bS✅ DONE — verified 2026-05-24: booking-engine renders all 9 presets (hmp-booking-engine/src/lib/themes/); property-service persists; resolve-host returns themePresetId; PB-2 (44006e3) shipped dashboard picker; PD-2 (76d527b) hardened preview iframe. Skill doc rewrite ships in same batch (Task 2).
AP-T03-004Verify Razorpay creds + webhook URL on QA + prodS🟡 PARTIAL (3b555db audit(AP-T03-004); need prod LIVE verification per AP-T12-002)
AP-T04-003Wire guestName in search() via GuestServiceClientS✅ DONE (verified 2026-05-23 already-shipped — c7a8acb + 3d90b39)
AP-T04-004Replace client-side FolioTable with rows from res.folios[].transactions[]M✅ DONE 2026-05-23 (4b58c2a — buildFolioViewModel; fixed a latent tax double-count bug)
AP-T05-004"Move Room" + "Extend Stay" CTAs on /reservations/:idS✅ DONE 2026-05-23 (abb8be5 — in-place dialogs)
AP-T05-005Consolidate RoomOpsService.roomMove + CheckInService.roomMoveM✅ DONE 2026-05-23 (8204aa3 — RoomOpsService.roomMove now canonical)
AP-T05-006Post folio CHARGE for paid-upgrade rate-difference inside room-move txS✅ DONE 2026-05-23 (8b17553 — atomic + idempotent)
AP-T06-005Switch ID-photo + signature uploads from base64 to /uploads/presigned-urlM✅ DONE 2026-05-23 (frontdesk self-checkin de998ff + booking-engine UI 92e2f53)
AP-T06-009Boot-time check: fail fast if ENCRYPTION_KEY unset in prodS✅ DONE (e925c04 fix(dev): add ENCRYPTION_KEY dev placeholder (AP-T06-009 boot gate))
AP-T07-004Extend FOLIO_CHARGE_POSTED payload with CGST/SGST/IGST/customerStateM✅ DONE — producer already-shipped (099b3ea); consumer 2026-05-23 (3054c0f — also shipped a MISSING FolioTransaction GST-columns migration)
AP-T07-005Introduce KitchenStation model + per-item routingL🔴 NOT STARTED
AP-T07-006Map transactionCode from orderType (ROOM_SERVICE/CANTEEN/etc)S✅ DONE — verified 2026-05-24: mapOrderTypeToTransactionCode() at hmp-pos-service/src/orders/events/folio-charge-posted.event.ts:85-92. Shipped 2026-04-29 in 099b3ea alongside AP-T07-004.
AP-T08-006Promote irn/signedQRCode/ackDate columns onto FolioS✅ DONE — verified 2026-05-20: Folio has irn/signedQRCode/ackDate/ackNumber (schema.prisma:49-52, migration 20260429120000_add_folio_irn_columns); EInvoiceJobProcessor writes them on COMPLETED. Shipped alongside AP-T08-001.
AP-T08-007Route credit-note IRN through EInvoiceJob queue with retry/DLQS✅ DONE 2026-05-23 (e8c4bcd)
AP-T08-008Sync the SELLER property GSTIN into the frontdesk read-model (e-invoice "Property GSTIN is required" 400 on real properties)S✅ DONE 2026-06-06 — two-hop read-model gap surfaced by the Phase-C real-IRN E2E. Producer: hmp-property-service property.created/property.updated now emit gstin/panNumber/addressLine1/phone. Consumer: frontdesk ReadModelSyncService.handlePropertyUpsert writes gstin/state/city/addressLine1/phone (conditional spread; partial events never NULL). Backfill existing rows via no-op PUT /properties/:id. Closes the seller-GSTIN half of D-005A-009 (buyer half = AP-T08-003/#5176). Specs: frontdesk read-model-sync.service.spec.ts (+2), property-service properties.service.spec.ts (+2).
AP-T08-009Register NIC webhook callback URL during sandbox onboardingS🛑 BLOCKED (depends on AP-T08-002 NIC creds)
AP-T09-003Implement night-audit-summary PDF/CSV generatorM✅ DONE 2026-05-23 (094e618 — CSV half + controller endpoint; PDF was already shipped)
AP-T09-004Add typed currentBusinessDate Date? column on PropertyM✅ DONE (a7169e7 bundled with T02-005/006)
AP-T09-005Promote NA scheduler to k8s CronJob with miss-detection alertS✅ DONE (8e724b7 audit(AP-T09-005+AP-T12-005): night-audit CronJob)
AP-T10-006Subscribe to reservation.modified, fix folio.charge.posted→folio.posted drift, add folio.closed/kot.completedM✅ DONE 2026-05-23 (c0d9160 — added pos.kot.completed; rest already shipped). NB: see FU-55 — frontdesk still never publishes folio.posted
AP-T10-007Replace /reports/[category] fallback list with backend catalogS✅ DONE — verified 2026-05-24: dashboard c9b045f (delete fallback) + 3fb2c15 (delete legacy page; /reports 308→/reports/v2); backend catalog 8b6e91e ships ReportRegistry with 36 auto-registered reports.
AP-T10-008Replace admin /analytics demo data with real APIM✅ DONE — verified 2026-05-24: hmp-admin-portal 6f91f91 (2026-04-29) — useAdminPlatformOverview() + GET /api/analytics/admin/platform-overview (cross-tenant, SUPER_ADMIN-only, 5-min cached); 6/6 specs.
AP-T10-009Fix MvDailyOccupancy.totalRooms upsert from PropertyServiceClientS✅ DONE 2026-05-23 (8e82d22 — sourced from PropertyServiceClient, self-heals on update)
AP-T11-004Implement MinLOS/MaxLOS push (OTA_HotelBookingRuleNotifRQ)M✅ DONE — verified 2026-05-24: shipped 1155b7f (2026-04-29) then redesigned 2d887b4 (2026-05-04) per ResAvenue request — canonical path is <MinStay>/<MaxStay> inside <Rate>, NOT inventory <LengthsOfStay>. Authoritative in hmp-channel-service/CLAUDE.md.
AP-T11-005Add flow-tests/flow-11-channel-manager.spec.ts end-to-endM✅ DONE (reconciled 2026-06-04)
AP-T11-006Set CHANNEL_MODE=simulator in docker-compose.qa-aws.ymlS✅ DONE (reconciled 2026-06-04)
AP-T11-007channelCodeMap + whitelistedIps for ResAvenue sandbox seedS✅ DONE (6e22ac9 audit(AP-T11-007))
AP-T11-016Spec + plan reconciliationS✅ DONE (eb5f0e4)
AP-T11-023ResAvenue cert cycle flow testM✅ DONE (b7104db audit(AP-T11-023))
AP-T11-024Seed default IntermediaryProviderConfig rowsS✅ DONE (9a42ad5 audit(AP-T11-024))
AP-T12-003Add payment_capture: 1 to Razorpay order bodyS✅ DONE — razorpay-gateway.adapter.ts:106 + idempotency header fix; 27/27 specs pass; verified 2026-05-20
AP-T12-004Build WebhookReconcileService cron (every 15 min)M✅ DONE 2026-05-23 (e6740bf — new PaymentReconcileService, root-cause layer for never-delivered Razorpay webhooks)
AP-T12-005Override HARDWARE_MOCK=false in docker-compose.prod-aws.ymlS✅ DONE (8e724b7 + 4ff2eb7)
AP-T13-003Multi-room reservation visual continuityM✅ DONE (reconciled 2026-06-04)
AP-T13-004Add mark-no-show to BAR_ACTIONSS✅ DONE (204d370 + (in 4ab98ee))
AP-T13-005Replace window.location.href redirect with in-place dialogS🟠 PARTIAL (reconciled 2026-06-04)
AP-T13-006Date-range block UI (RoomBlock model or HK MaintenanceRequest)M✅ DONE — verified 2026-05-24: duplicate-tracking of AP-T13-003 P0 row. RoomBlock model (hmp-reservation-service/prisma/schema.prisma:751-769) + RoomBlocksStaffController + block-range-dialog.tsx (15 specs) shipped 275b156 (2026-05-08) + bulk picker 1ce842e.

P1 closure summary (updated 2026-05-23): ✅ ~27 done (+12 closed in the 2026-05-23 consolidation batch: AP-T01-007, AP-T04-003/004, AP-T05-004/005/006, AP-T06-005, AP-T07-004, AP-T08-007, AP-T09-003, AP-T10-006, AP-T10-009, AP-T12-004) · 🟡 2 partial · 🔴 ~9 not started (AP-T07-005/006, AP-T10-007/008, AP-T11-004/005, AP-T13-003/005/006) · 🛑 1 blocked · ❓ 0 verify

4. Property #1 audit — P2 (39 items)​

P2s abbreviated per source audit. Status: presumed 🔴 NOT STARTED for most unless evidence found in git log. Detailed enumeration not done here — refer to property-1-readiness-audit-2026-04-28.md Section E "P2".

Includes (selected): Google Maps geocoding, amenity master catalog, multi-property booking-engine selector, multi-guest C-Form per foreign co-traveller, NLQ wiring, MV nightly self-heal, ESC/POS receipt printing in checkout, bulk-block toolbar, mobile responsive toolbar, drag/drop Playwright e2e, root↔per-folder CLAUDE.md drift cleanup, frroCity column, RatePlanRate seed back-fill.

P2 disposition for Phase 1: Likely all DESCOPED. Schedule a separate "post-launch hardening" plan to triage what's actually wanted for Phase 2.

5. Plan 4/5/6/7 status​

Plan 4 — Demo tenant on prod​

TaskStatusAction
Task 17 in-cluster seed-runner Job✅ DONEUsed by auto-seed pipeline
Task 19 read-model fanout (5 services)✅ DONEConfirmed in code
Task 20 @hmp/api-contracts@1.10.0 + dashboard yellow [DEMO] chip✅ DONEVerified 2026-05-23 — @hmp/api-contracts@1.12.2, simulation-badge.tsx present
Tasks 21-23 e2e seed smoke runs (test/qa/staging)✅ DONEAuto-seed pipeline handles
Task 24 prod demo-tenant seed🛑 BLOCKED (reconciled 2026-06-04) — Job authored + runbook; awaits 2-of-N approvalOperator: requires Q4 on-call rotation set up first
Plan 4-bis Asset-mgmt demo enrichment (Phases 1-3)✅ DONE 2026-05-18..20All 4 envs

Plan 5 — Cutover​

ItemStatus
api.keynova.ai + live.keynova.ai Ingresses✅ Live on prod EKS ALB (2026-05-17)
ResAvenue PROD handler verified end-to-end✅ Returns proper OTA-XML 400
7-day prod burn-in observability🟠 PARTIAL (reconciled 2026-06-04) — Day 3 of 7 (closes 2026-05-24)
FU-6 ResAvenue cert v3 closure🟠 PARTIAL (reconciled 2026-06-04) — Closes after burn-in completes
ResAvenue first hotel onboarding✅ DONE (reconciled 2026-06-04) — Runbook ready — hmp-infrastructure/monitoring/runbooks/resavenue-prod-go-live.md (item E1/E2, §12). Blocked on the per-hotel ResAvenue creds (hotelCode/RequestorID/password).

Plan 6 — Prod-launch baseline​

ItemStatusAction
Baseline checklist YAML executed once✅ 2026-05-17 (1 PASS / 7 FAIL / 13 NOCREDS / 2 AUTOMODE)
Baseline RE-run after AWS profile + flag fixes🔴 NOT DONEOperator
B-17 Renovate.json propagation (25 repos)🟡 Template + script authoredOperator runs scripts/propagate-renovate-config.sh
B-18 First quarterly access review (2026-Q2)🟡 Runbook + schedule shippedFirst review due 2026-06-30
B-19 DR drill (failover ap-south-1 → ap-south-2)🔴 NOT EXECUTEDSRE
B-20 Backup-restore drill against keynova-prod-vault🔴 NOT EXECUTEDSRE
B-21 Pentest report🛑 VENDOR NOT ENGAGED (2-4w lead time)External
B-23 CIC-1..5 sign-offs🛑 ALL signed=0Product/Security/SRE/CS/Payments
B-24 Spec §14 5 decisions (Q1-Q5)🟠 PARTIAL (reconciled 2026-06-04) — Recommendations draftedQ4+Q5 are hard blockers

Plan 7 — Legacy retirement​

PhaseStatus
Phase 1 inventory + 2-of-N approval✅ Authored
Phase 2 data-snapshot✅ Skipped per operator (seed-only data)
Phase 3 destroy (legacy account 730335256638)🟢 IN PROGRESS — operator manually deleting via Console (3+ days)
Phase 4 code cleanup (git rm legacy Terraform stacks + compose + scripts)🟡 PENDING Phase 3 — legacy dirs terraform/{pilot-aws,staging-aws,prod-aws} still exist
Phase 5 cost verification (T+7d post-destroy)🟡 PENDING
FU-3 legacy module dedup🟡 OPEN (reconciled 2026-06-04 — Plan 7 Phase 4 not yet committed)

6. Follow-ups (FU-2 .. FU-53)​

Cross-referenced from hmp-infrastructure/compliance/follow-ups.md (1005 lines).

Resolved (✅)​

  • FU-1 Trufflehog secret scanning ✅ (Plan 3 T30)
  • FU-3 Legacy module dedup 🟡 OPEN (reconciled 2026-06-04 — Plan 7 Phase 4 not yet committed)
  • FU-4 region-lock SCP IAM exemption ✅
  • FU-7 Kyverno disallow-aws-credentials-env rewrite ✅ (2026-05-11)
  • FU-8 vpc-cni ENI pod cap ✅ (Plan 2 T12)
  • FU-9 EKS bearer token exec-auth ✅
  • FU-10 pod-security duplicate Namespace ✅
  • FU-15 RDS logical DBs Job ✅
  • FU-16 Loki PVC immutability ✅
  • FU-17 Loki helm timeout ✅
  • FU-44 @hmp/cache + @hmp/messaging TLS support ✅
  • FU-47 @hmp/observability HealthService paramtypes ✅
  • FU-50 promote-image.sh rebase-retry ✅
  • FU-52 RDS auto-rotation breaks pods (Reloader) ✅ PROD-VALIDATED 2026-05-24 — overnight cron(0 21 ? * SAT *) fired 03:41:21 IST Sunday; ESO+Reloader restarted all 17 RDS-touching backends within ~34s; cadence reset to AWS-managed weekly (AutomaticallyAfterDays=7, NextRotation 2026-05-31T05:29:59+05:30). Evidence: [[project_2026-05-24-fu-52-prod-validated]].

Open​

IDItemSeverityOwnerStatus
FU-2Tooling installation runbook (tflint/tfsec/checkov local)LowPlan 0 carry-over🟠 PARTIAL (reconciled 2026-06-04)
FU-5AWS Config delivery channel + S3 Object Lock incompatibilityMediumPlan 6🟡
FU-6ResAvenue cert v3 closureMediumCloses 2026-05-24🟢 IN PROGRESS
FU-11IDC OIDC ArgoCD console registrationLowOp🟡
FU-12ArgoCD per-env PAT populationLowOp🟡
FU-13Windows DNS cache poisoningLowWorkaround🟠 PARTIAL (reconciled 2026-06-04)
FU-14Helm provider v3 Windows TCP flakesLowWorkaround🟠 PARTIAL (reconciled 2026-06-04)
FU-18Kustomize v5 namespace transformer ID conflictMediumArchitecture🟠 PARTIAL (reconciled 2026-06-04)
FU-19IDC group→permission-set excludes mgmt accountLowOp🟡
FU-20ArgoCD SSO needs SAML 2.0 refactor (~1d)HIGHOp🔴 OPEN — actively blocking prod ArgoCD operator access. Surfaced 2026-05-30 during FU-30 prod rollout: clicking "Login via Identity Center" on argocd-prod.keynova.ai returns failed to query provider "https://943273444105.signin.aws.amazon.com/oidc": ... expected Content-Type = application/json, got "text/html". Root cause: the URL is an AWS Console signin page (HTML), not a valid OIDC issuer; AWS IDC doesn't expose itself as an OIDC provider at per-account signin URLs. Workaround used 2026-05-30: bootstrap admin password retrieval via `kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}'
FU-21Bot PAT stored in mgmt account; ESO IRSA env-localLowOp🟠 PARTIAL (reconciled 2026-06-04)
FU-22.gitattributes did not cover .tf filesLowResolved (commit 2026-05-11)✅
FU-23Mgmt-account Azure DevOps OIDC + deploy roleMediumResolved via gap-fill stack✅
FU-24Hotfix pipeline assumes Jira; we don't have oneMediumPlan 6🟡
FU-25Migrate deploy notifications Slack → TeamsLowQ3 decision🟡
FU-26ArgoCD ci-bot tokens non-expiring + no revocation runbookMediumPlan 6🟡
FU-27ArgoCD helm_release timeout false-negativeLowWorkaround🟡
FU-28Self-approval allowed on staging/prod during pilotMediumCloses when Q4 lands🟡
FU-29Plan 3 T27/T28 per-service overlay subdirectoriesMediumResolved (Plan 3 SHIPPED)✅
FU-30Per-service ServiceAccounts missing IRSA annotationsMediumPlan 6🟢 PARTIAL — property-service ✅ FULLY DONE all 4 envs 2026-05-30 (test/qa/staging/prod all terraform apply'd cleanly via scripts/fu-30-apply-property-assets.sh + ArgoCD sync'd + smoke-tested through dashboard UI for upload + list + display + delete); notification-service SES ✅ DONE 2026-05-21 (OP-2); auth-service basic SA ✅; remaining 🟡: frontdesk (ID photos, see FU-68), guest (signatures, see FU-69), payment (Razorpay/SecretsManager). Property-service close-out: terraform/modules/assets-storage/ (S3 bucket keynova-<env>-assets + IRSA role keynova-<env>-property-assets); per-env Kustomize overlay SA annotation + AWS_S3_BUCKET ConfigMap patch; operator runbook monitoring/runbooks/fu-30-property-assets-irsa-apply.md; one-shot apply script scripts/fu-30-apply-property-assets.sh (idempotent, per-env SSO + mgmt-account check, plan-aborts-on-destroy, 4 polish iterations). Buckets: keynova-{test,qa,staging,prod}-assets. Roles: arn:aws:iam::<env-account>:role/keynova-<env>-property-assets. Property #1 admin's original SEV-1 is resolved. Two follow-up blockers discovered + fixed + verified during the rollout: FU-76 (dashboard CSP) + FU-77 (private-bucket-needs-presigned-GET).
FU-32Plan 3 templates trigger:/pr: blocksLowResolved✅
FU-43Per-env auth-service-runtime-env propagationMediumOp runtime secret rotation🟠 PARTIAL (reconciled 2026-06-04)
FU-45Service runtime secret rotation policy + LambdaMediumPlan 6🟡
FU-46RS256 JWT keys for staging+prod (also hard blocker §1)HIGHOp + Eng✅ DONE 2026-05-21 — all 4 envs, see §1
FU-48Asset-service migration history resetMediumOp playbook ready🟠 PARTIAL (reconciled 2026-06-04)
FU-49Notification-service device_tokens orphan migrationMediumSame playbook as FU-48🟠 PARTIAL (reconciled 2026-06-04)
FU-51Real SendGrid prod key (also hard blocker §1)HIGHOp🛑 BLOCKED — superseded by SES (AP-CC-6 / board #6454) (reconciled 2026-06-04)
FU-53metrics-server missing on test+qaMediumOp + Eng✅ DONE 2026-05-20 — Operator ran helm upgrade --install metrics-server metrics-server/metrics-server v3.13.0 on ALL 4 envs (turned out staging+prod were also missing, not just test+qa). All 4 envs now show metrics-server-c978df89c-* 1/1 Running and kubectl top nodes returns CPU/memory. Long-term Terraform addon at terraform/modules/eks-addons/metrics-server.tf ready for next IaC reconciliation.
FU-54QA POS + payment producer demo data — backfill events so MV pos_revenue / payment_collection / einvoice_register / refund_register / settlement_register populateMediumOp + Eng🟡 OPEN 2026-05-20 — surfaced by AP-T10-003 diagnosis. Plan 4 demo enrichment covered reservation/folio/housekeeping/channel surfaces but did NOT cover POS orders, payment captures, or e-invoice IRN generation in QA. Either extend the demo seed to publish these events OR accept these 5-7 MVs as empty until a real POS/payment flow lands.
FU-55Plan-2 producer-side field emission — services must include segment/countryCode/source/roomTypeCode/ratePlanCode/assignedTo in event payloads so 7 secondary MVs populateMediumEng (per producer service)✅ DONE 2026-05-23 — all 3 producer sides shipped. reservation-service (866391d — segment/countryCode/source/roomTypeCode/ratePlanCode on reservation events) + housekeeping-service (e8bf812 — assignedTo already emitted, regression-locked) + frontdesk-service (208e25d merged to master ad2d036 — dual-emit FOLIO_POSTED + FOLIO_CLOSED alongside legacy FOLIO_CHARGE_POSTED + FOLIO_SETTLED, with analytics-segmentation fields: segment/countryCode/roomTypeCode/ratePlanCode/roomNights/taxAmount, all fail-soft via PropertyServiceClient + GuestServiceClient resolvers. Pipeline #30824). The empty mv_revenue_by_segment / mv_country_stats / mv_daily_sales_register / mv_room_type_revenue / mv_rate_plan_performance MVs will populate from the next folio events posted post-deploy. Next-Track-4 cleanup: retire the deprecated folio.charge.posted + folio.settled topics after one release cycle of analytics CIs running green. Original: surfaced by AP-T10-003 diagnosis.
FU-56mv_revenue_by_source anomaly — analytics MV has 140 rows from an unknown source path. FU-55-frontdesk investigation (agent ae14b4d 2026-05-23) confirmed it is NOT from FolioConsumer (which never fired before today's dual-emit fix). Could be legacy backfill, a different consumer in a different service, or a direct DB write. Read-only investigation only; informational.LowEng (read-only)✅ RESOLVED 2026-05-24 — not an anomaly. ReservationConsumer.onReservationCreated (hmp-analytics-service/src/consumers/reservation.consumer.ts:188) upserts mv_revenue_by_source on every reservation.created event; has been live since the service shipped (pre-FU-55). 140 rows = expected. FU-55's dual-emit only affected the FOLIO path.
FU-57Test-rot audit for the 9 backend services NOT covered in 2026-05-23 cleanup — hmp-banquet-service, hmp-channel-service, hmp-crm-service, hmp-pos-service, hmp-asset-service, hmp-auth-service, hmp-property-service, hmp-spa-service, hmp-housekeeping-service. The 5 services audited 2026-05-23 yielded 187 test bugs / 0 product bugs — pattern likely repeats.MediumEng🟡 OPEN 2026-05-23 — pattern: DI drift after service-constructor grows, missing mock delegates, stale assertions vs evolved canonical contracts, wrong test inputs. Diagnostic: per-repo npx jest --testPathIgnorePatterns=e2e and npx tsc --noEmit.
FU-58CI infrastructure for env-dependent e2e specs — ~185 reservation-service specs + frontdesk roundtrip.e2e.spec.ts hardcode localhost:5438/localhost:6379. They fail outside a local-dev environment with Postgres+Redis up. Move to a docker-compose-based e2e harness in CI, OR mark with a .e2e.spec.ts suffix and exclude from unit-test runs.MediumOp + Eng🟡 OPEN 2026-05-23 — surfaced by 2026-05-23 test-rot residual triage agent. Blocks meaningful AP-T16-004 coverage measurement on these flows.
FU-59Establish periodic spec-contract-refresh practice (quarterly) — 187-spec rot uncovered 2026-05-23 came from services intentionally evolving (new constructor deps, new contracts, new async timing) without specs being refreshed. A quarterly npx jest && npx tsc per repo with triage of any drift would catch this early.LowEng (process)🟡 OPEN 2026-05-23 — recommend a quarterly calendar reminder + wiki page tracking the per-service most-recent green test-run date.
FU-60Enable ArgoCD self-heal + add CI guard for base-vs-overlay drift — 2026-05-23 found 6 of 20 prod backends silently missing reloader.stakater.com/auto annotation because their per-service overlays hadn't been touched since base commit e66b694 (FU-52). ArgoCD's per-overlay change detection missed the upstream base change. Two complementary fixes: (a) enable selfHeal: true + prune: true on every Application so drift gets auto-corrected; (b) CI guard — when a kubernetes/base/apps/* manifest is modified, automatically touch every referencing overlay in the same PR.HIGHOp + Eng🟡 OPEN 2026-05-23 — would have prevented the 6-deployment annotation gap before it ever reached prod. Related memory: [[feedback_argocd-misses-base-changes-without-overlay-touch]].
FU-61api-gateway /api/metrics returns 404 — Prometheus scrapes the gateway at /api/metrics but the gateway doesn't serve metrics there (metrics are per-service, scraped directly). Pre-existing config quirk surfaced during Reloader prod-validation 2026-05-23. Either configure Prometheus to skip the gateway scrape, or expose /api/metrics at the gateway.LowEng (api-gateway)✅ DONE (reconciled 2026-06-04) — non-blocking; cosmetic 404 noise in Prometheus + api-gateway logs.
FU-62Retire deprecated folio.charge.posted + folio.settled topics after analytics CI is green for one release cycle on canonical folio.posted + folio.closed — frontdesk-service now dual-emits both (per FU-55-frontdesk fix 208e25d). Once dual-emit has run clean for ≥1 week, deprecated emits can be dropped (PosFolioChargeConsumer's subscription needs to switch to canonical first).LowEng (frontdesk + pos-folio-charge consumer)🟡 OPEN 2026-05-23 — earliest realistic retire date: 2026-05-30 (one week of dual-emit running clean).
FU-63AP-T06-002 check-in flow wiring — guest-service GuestVisa model + API shipped 2026-05-23 (06eb705 → master ee23667); frontdesk check-in flow still uses the legacy idType=OTHER storage path. Three pieces needed: (a) add GuestServiceClient.upsertGuestVisa() / .getGuestVisa() methods in @hmp/service-clients, (b) wire frontdesk check-in service to persist visa via guest-service, (c) dashboard check-in form: typed visa-fields section. ~3-4 days for a focused session.MediumEng🟡 OPEN 2026-05-23 — deferred from 2026-05-23 priority-20 batch. Unblocked now that guest-service GuestVisa branch is merged.
FU-64Investigate stuck-Terminating pods — during 2026-05-23 Reloader test on keynova-test, hmp-property-service-698fc6b5cd-tckc9 stayed in Terminating for 8m20s before kubelet cleaned it up (way past the 60s terminationGracePeriodSeconds set in L1 hardening commit fb69a67). New replicas were Ready so it wasn't blocking, but indicates a hung preStop/finalizer or kubelet stall. Repeats across future rotations would surface a real issue.LowEng + Op🟡 OPEN 2026-05-23 — observed during Reloader prod-validation walkthrough. Could be related to the L1 preStop sleep 10.
FU-65Sunday-morning verification of overnight Reloader prod auto-rotation + schedule reset — 2026-05-23 scheduled prod RDS rotation cron(0 21 ? * SAT *) (window: 02:30–05:30 IST Sunday 2026-05-24). Morning checklist: confirm LastRotated timestamp updated, all 20 deployments still Ready, API smoke 200, Reloader logs show ~17 deployment restarts. Then reset schedule from cron back to AutomaticallyAfterDays=7 so it doesn't fire every Saturday going forward.MediumOp (Sunday 2026-05-24 morning)✅ CLOSED 2026-05-24 — rotation fired 03:41:21 IST; ESO syncedResourceVersion bumped 1-…→2-fdd81e25…; Reloader logged 17 Changes detected events at 03:41:55-57 IST; all 17 RDS-touching backends rolling-restarted by ~04:40 IST (verified 12:40 IST: 17 backends all at AGE 8h, frontends correctly skipped); API health 200 throughout. Cadence reset via rotate-secret --rotation-rules AutomaticallyAfterDays=7 --no-rotate-immediately — verified Duration=None, ScheduleExpression=None, LastRotated=2026-05-24T03:41:21+05:30, NextRotation=2026-05-31T05:29:59+05:30. Diagnostic gotcha logged: aws sso login --profile X does NOT change kubectl context — always kubectl config current-context first. Full evidence: [[project_2026-05-24-fu-52-prod-validated]].
FU-66Comprehensive branching + per-env pipeline policy — currently push-direct-to-master triggers all envs. Target: separate test/qa/staging/prod branches with environment-specific pipeline triggers. Once landed, upgrade FU-60(b) CI guard from informational-only to blocking.MediumEng + Op🟡 OPEN 2026-05-24 — surfaced during Next-20 brainstorm. Not blocking first-property launch; queued for post-launch hardening.
FU-67Codebase-wide audit of hardcoded limit values exceeding the shared PaginationDto.@Max(100) cap — surfaced 2026-05-25 when the channel-manager onboarding wizard's <TenantPicker> hit 400 Validation failed because useTenants hardcoded limit: 1000. Audit completed 2026-05-26: scanned hmp-admin-portal, hmp-web-dashboard, hmp-booking-engine, hmp-mobile-apps. 5 bugs fixed in hmp-web-dashboard commit 494673a: TeamWorkload tasks (limit: 1000 — guaranteed 400 on every load), Work-orders Kanban refresh (200→100), Stock balances (200→100), Stock movements (200→100), Vishramgruh authorizations enrichment (500→100). hmp-admin-portal verified clean (yesterday's Option B server-side search + cap-at-100 fix). hmp-booking-engine + hmp-mobile-apps clean (zero hits). v2 follow-ups noted: TeamWorkload needs a server-side per-staff aggregate endpoint, authorizations needs GET /authorizations/active-by-bed, Kanban needs filter chips for >100 work orders.HIGHEng (codebase-wide)✅ CODE-COMPLETE 2026-05-26 — hmp-web-dashboard commit 494673a; pipeline pending.
FU-68Frontdesk-service S3 IRSA — closes the second leg of FU-30 (guest ID photo uploads from the self-check-in kiosk + dashboard check-in flow). Reuses the FU-30 module's bucket (module.assets_storage.bucket_arn) — needs a NEW IRSA role keynova-<env>-frontdesk-assets trust-policied to system:serviceaccount:hmp-apps:hmp-frontdesk-service with the same S3 PutObject/GetObject/DeleteObject grants. Pattern is a copy-paste of the property-assets role in terraform/modules/assets-storage/main.tf + a per-env Kustomize overlay SA annotation. Until shipped, ID photo uploads still depend on the legacy AWS_ACCESS_KEY_ID env var which is unset in EKS (silently broken since Plan 1 migration).MediumEng🟡 OPEN 2026-05-26 — surfaced during FU-30 property-service closure. ~1 hour of work; mirror of the already-merged property-service pattern.
FU-69Guest-service S3 IRSA — closes the third leg of FU-30 (guest signature uploads from the registration card flow + DSAR export PDF). Same bucket as FU-30 + FU-68; new role keynova-<env>-guest-assets with the same scope. Same one-hour copy-paste of the property-assets pattern.MediumEng🟡 OPEN 2026-05-26 — surfaced during FU-30 property-service closure.
FU-70@hmp/api-contracts rate-plan schema cleanup — createRatePlanSchema.validFrom / validTo currently declared as z.string().datetime() which rejects YYYY-MM-DD (the output of HTML <input type="date">). Two dashboard hotfixes (2026-05-26 a9cee61 for ISO format + 4a32de2 for null→undefined) work around the schema by transforming dates before validation and using undefined (not null) for missing optionals. Cleaner long-term fix: change schema to z.string().date().optional() AND optionally .nullable().optional() for fields the dashboard wants to explicitly clear (derivedFromId / derivationMethod / derivationValue / maxStay / maxAdvanceBooking). Requires bumping @hmp/api-contracts minor (e.g. 1.13.0 → 1.14.0), publishing to Azure Artifacts, then bumping the dashboard's package.json dep. The workaround code can then be removed in a follow-up cleanup.LowEng🟡 OPEN 2026-05-26 — schema fix unblocked by workarounds; cleanup-only, no user-visible bug.
FU-71Dashboard || null sweep — the rate-plans form bug (expected string, received null from zod's .optional() not accepting null) is almost certainly NOT isolated. Many dashboard forms use field: input || null as the empty-fallback pattern against zod-validated payloads where the schema is .optional() (stringundefined) not .nullable(). Audit: grep -rn '|| null,' hmp-web-dashboard/src/app | grep -v '__tests__|test\.'. Triage each match against its target schema (most consumers import from @hmp/api-contracts). Fix pattern: use || undefined for .optional() fields, or switch the schema to .nullable().optional() if the form genuinely needs to send "clear this field" semantics (paired with FU-70 type cleanups).MediumEng
FU-72Test coverage gap on hmp-web-dashboard/src/app/(dashboard)/settings/rate-plans/page.tsx — zero vitest specs exist for this page despite two production-blocking bugs in 24 hours (ISO date + null-vs-undefined). The create-plan happy path with a date filled in would have caught both. Add specs that exercise: (a) create with all fields filled (asserts ISO datetime is sent + null fields are absent from the payload), (b) create with optional fields blank (asserts undefined is sent or fields omitted, NOT null), (c) edit flow with hydrated values, (d) derived-plan toggle. Add similar specs for the other major settings pages (room-types, taxes, cancellation-policies, season-templates) — same anti-pattern likely present.MediumEng🟡 OPEN 2026-05-26 — production bugs shipped because nothing exercised these paths. Track per-page coverage in the rolling spec audit (FU-57).
FU-73Dashboard pipeline auto-trigger consistently fails on master pushes — every git push origin master on hmp-web-dashboard requires a follow-up manual az pipelines run --id 394 --branch master to actually start the build. Repeats across at least 3 sessions (PR-2 FU-30-followup, rate-plan ISO fix, rate-plan null fix). The trigger config in the Azure pipeline is present and master is in the trigger branches; this is an Azure DevOps platform-level issue, NOT a pipeline YAML bug. Operator workaround is reliable but adds 30-60s of friction per push. Investigation: check the pipeline's Run for triggered builds setting in Azure DevOps UI; verify the service connection's webhook is actually subscribed; check Azure DevOps Pipelines status page for known throttling. Known issue per memory feedback_prefer-merge-to-master-over-per-branch-ci. Affects all 24 service pipelines (occurs but only systematically observed on dashboard so far).MediumOp + Eng🟡 OPEN 2026-05-26 — workaround is az pipelines run; need an audit pass on the Azure DevOps webhook subscriptions for all 24 pipelines, plus possibly a settings.json or YAML annotation tweak.
FU-74Property Settings UX consolidation — /settings/property has 12 tabs (Profile, Contact & Address, Operations, Amenities, Policies, Operational Policies, Payment, Tax & Compliance, Registration, Highlights, Extra Charges, Photo Gallery) which is cognitive overload for first-time admin users. Three discrete fixes surfaced during the FU-30 Photo Gallery review: (1) Setup Wizard ↔ Property Settings cross-link — Property Settings loading subtitle says "For initial setup, use the Setup Wizard" but the rendered state replaces it with the generic "Configure your property details and preferences". Add a persistent "→ Use Setup Wizard for first-time setup" link on the rendered state, AND a "→ For ongoing edits, use Property Settings" cross-link on the wizard. (2) Merge "Policies" + "Operational Policies" tabs — both contain child-age / charge fields stored under different settings paths (settings.policies.* vs settings.operationalPolicies.childPolicy.*); the labels give users no hint of which goes where. Either merge into a single Policies tab with sub-cards, or rename to disambiguate (e.g. "Property Rules" vs "Charge Policies"). (3) Tab count reduction — consolidate to ~6 logical groups (Profile, Location & Contact, Operations & Policies, Tax & Registration, Photos, Extras) or switch to a sidebar nav.LowUX + Eng🟡 OPEN 2026-05-26 — surfaced during FU-30 Photo Gallery review. Non-blocking; quality-of-life improvement.
FU-76✅ FULLY VERIFIED test/qa/staging/prod 2026-05-30. Dashboard CSP blocked direct S3 uploads + leaked pod-internal authority from request.nextUrl.origin in EKS standalone mode. Surfaced during qa smoke test of FU-30 on 2026-05-30: presigned URL from property-service IRSA generated correctly (real X-Amz-Security-Token visible) but browser blocked the PUT with connect-src 'self' https://0.0.0.0:3020 wss://0.0.0.0:3020. Two issues fused into one CSP header: (a) no S3 bucket origin allowed in connect-src, (b) request.nextUrl.origin in Next.js standalone mode with HOSTNAME=0.0.0.0/PORT=3020 (per ConfigMap) reflects the pod's bind address rather than the public ALB hostname, leaking into both the apiOrigin and wsOrigin slots. Same-origin WSS (Socket.IO notifications) was also broken because of (b). Fix shipped 2026-05-30: hmp-web-dashboard/src/middleware.ts (a) adds https://*.s3.ap-south-1.amazonaws.com to connect-src — wildcard covers all 4 env buckets, (b) reconstructs the public origin from x-forwarded-host → host header → nextUrl.host with an explicit reject list for the untrusted authorities (0.0.0.0, localhost, 127.0.0.1). Two commits because the first attempt only filtered NEXT_PUBLIC_API_URL (which was already empty), missing that requestOrigin itself was the leak source: 62d09c7 (initial) + 9210158 (fix-on-fix from header reconstruction). Build 31484 promoted 9210158 to all 4 env overlays; auto-synced on test+qa, manual sync pending for staging+prod. Test smoke verified upload works. (Renamed from FU-75 on 2026-05-30 because that ID was already claimed by an unrelated 2026-05-26 Active Connections shipment.)HIGHEng🟢 SHIPPED 2026-05-30 — test+qa verified; staging/prod manual sync pending.
FU-77✅ FULLY VERIFIED test/qa/staging/prod 2026-05-30. Property images don't display in the dashboard gallery after upload — the <img src={img.url}> 403's against the bucket. Surfaced 2026-05-30 during test smoke right after FU-76 unblocked the upload itself. Root cause: the FU-30 terraform/modules/assets-storage/ provisions keynova-<env>-assets with all four S3 public-access blocks ON (block_public_acls / block_public_policy / ignore_public_acls / restrict_public_buckets) — intentionally, since the same bucket will hold guest ID photos / signatures (FU-68/69) which MUST stay private. But AssetService.listImages / registerImage returned each row with url set to the canonical https://keynova-<env>-assets.s3.ap-south-1.amazonaws.com/<objectKey> (via getBaseUrl()), which 403's against the locked-down bucket. Bug was masked in dev because MinIO defaults to public reads, and the 2026-05-25 test smoke only verified bucket+SA existence — never actually rendered an image tile through the UI. Fix shipped 2026-05-30 (hmp-property-service 6a36cbd): new withPresignedGetUrl(row) helper generates a 1-hour presigned GET URL via GetObjectCommand + getSignedUrl keyed on objectKey; listImages calls it once per row (Promise.all); registerImage calls it on the just-created row so the dashboard's optimistic update has a working preview. Defensive: preserves stored URL when objectKey is missing (legacy rows). Stored url column unchanged — kept as the canonical reference for a future CloudFront/OAC migration where the URL pattern is stable; every read overrides it. 3 new specs + 27 existing specs all pass. Build 31487 promoting through. Do not relax bucket public-access blocks — security regression; CloudFront with OAC is the right long-term answer (tracked as FU-78).HIGHEng🟢 SHIPPED 2026-05-30 — pending build 31487 promote + re-smoke on test.
FU-78CloudFront + OAC in front of keynova-<env>-assets for production-grade public asset delivery. Today, FU-77 returns 1-hour presigned GET URLs from property-service on every listImages read — works, but every dashboard image tile requires a backend roundtrip + a fresh signature. Better long-term: a CloudFront distribution per env with Origin Access Control (OAC) signing requests to the private bucket, serving images at https://assets-<env>.keynova.ai/<objectKey> (or similar). Properties: cacheable at the edge, stable URL pattern (no signature in URL), CORS centrally configured, optional WAF. Booking-engine guest-facing image consumption (room photos in search results) will need this anyway. Scope: distribution + cert (*.keynova.ai wildcard already exists per Plan 0) + ALIAS + S3 bucket policy granting CloudFront's distribution principal. Once shipped, AssetService.getBaseUrl() can return the CloudFront URL and withPresignedGetUrl becomes a no-op.LowEng (post-launch)🟡 OPEN 2026-05-30 — surfaced during FU-77. Non-blocking for Property #1; queued for post-launch hardening.
FU-75Active Connections edit/delete UX shipped — closes the gap surfaced during Lotus Leaf prod onboarding where the /channel-providers/[code] Active Connections tab was read-only. hmp-channel-service adds DELETE /intermediary/connections/:id (SUPER_ADMIN, 409 if isActive=true, transactional cascade to IntermediaryMapping + ChannelRoute). hmp-admin-portal adds connection detail page (/channel-providers/[code]/connections/[connId]), inline actions on the connections tab (Manage mappings / Edit / Deactivate / Activate / Delete), and <DeleteConnectionDialog> with type-to-confirm UX. Two-step delete gate (Deactivate → Delete) + cascade-count disclosure. Backend commits: hmp-channel-service 337d7f2/93b39c4/3dd7ecd (pipeline #381). Frontend commits: hmp-admin-portal f9bb86e/96cb879/2148e26/0eb84e3/95abd85/bb7951f/53f0e7f (pipeline #30983). 7 backend + 24 frontend new test cases. Spec: hmp-docs/docs/superpowers/specs/2026-05-26-active-connections-edit-delete-ux-design.md. Plan: hmp-docs/docs/superpowers/plans/2026-05-26-active-connections-edit-delete-ux.md.LowEng✅ DONE 2026-05-26 — pipelines #381 + #30983 in flight; staging+prod promote-approvals pending.
FU-76Onboarding wizard cascade-creates ChannelRoute + ChannelConnection placeholder rows alongside the IntermediaryConnection. Surfaced 2026-05-26 during Lotus Leaf prod onboarding: wizard's POST /intermediary/connections was creating only the IntermediaryConnection row, not the 5 ChannelConnection placeholder rows or the 5 ChannelRoute (INTERMEDIARY routeType) rows that SyncService.pushRates/pushAvailability need to route outbound traffic through the intermediary adapter. Lotus Leaf workaround (one-time): hand-seed at hmp-infrastructure/scripts/one-off/lotus-leaf-seed-routes.yaml. Code fix: hmp-channel-service commit e2fb994 — IntermediaryService.createConnection wraps the insert + N× channelConnection.createMany + N× channelRoute.createMany (one per dto.supportedChannels[], routeType=INTERMEDIARY, intermediaryId=<new>, skipDuplicates=true) in a single prisma.$transaction. updateConnection cascade-syncs the supportedChannels delta (added → createMany; removed → updateMany isActive:false to preserve audit trail). deleteConnection already cascades correctly (FU-75). 7 new test cases (4 cascade + 3 delta). Pipeline #31030 in flight; staging+prod approvals pending. Hotel #2 onboards end-to-end with zero manual SQL once this + FU-78 (#31028) both reach prod.MediumEng✅ CODE-COMPLETE 2026-05-26 — pipeline #31030 in flight; staging+prod approvals pending.
FU-77Codebase-wide Prisma schema-vs-migrations drift audit. Surfaced 2026-05-26 when /distribution/ari 500'd with The table public.rate_restrictions does not exist. Audit completed 2026-05-26: scanned all 16 backend services (13 core + spa + asset + revenue-strategy). 60 drift sites found across 7 services; each fixed with an idempotent CREATE TABLE IF NOT EXISTS catch-up migration. Commits: hmp-spa-service fb49cad (3 tables), hmp-reservation-service 926f387 (7+col), hmp-guest-service 635821a (17 tables — largest drift), hmp-frontdesk-service 1d99618 (11), hmp-channel-service 3c9d67c (9), hmp-notification-service c6c1e0b (10), hmp-property-service 35f4134 (3 incl. the original rate_restrictions from FU-77 trigger). 9 services verified clean: auth/housekeeping/pos/banquet/crm/payment/asset/revenue-strategy/analytics (mv_daily_kpi false positive — handled via ALTER TABLE RENAME in 20260501032909_plan-2-mvs). Root cause across all sites: dev workflow leaning on prisma db push rather than prisma migrate dev — knew migrations were skipped (CLAUDE.md Gotcha #11 in guest-service explicitly said so). All 7 migrations use IF NOT EXISTS + DO-block enum guards + DO-block FK guards, so replay against envs that already have the tables (hand-applied via Job, or db push-ed in dev) is a no-op. 7 pipelines triggered.HIGHEng (codebase-wide)✅ CODE-COMPLETE 2026-05-26 — 7 migrations shipped across 7 services; pipelines pending.
FU-78Wizard backend hydrates endpoint + whitelistedIps from provider catalog at IntermediaryConnection create time — surfaced 2026-05-26 when Lotus Leaf's first rate push failed with Failed to parse URL from undefined. Root cause: hmp-admin-portal onboarding wizard's POST /api/intermediary/connections intentionally omitted platform-tier fields (endpoint, whitelistedIps, webhookSecret) from the credentials JSON — per the 2026-05-25 spec they were expected to be hydrated server-side from the IntermediaryProviderConfig catalog. But that resolver path is gated by INTERMEDIARY_PROVIDER_TABLE_ENABLED (default OFF), so the connection's credentials.endpoint ended up undefined. Node fetch(undefined) then threw the error on every outbound push. Hot-patched for Lotus Leaf via kubectl exec into the channel-service pod (decrypt → add endpoint → re-encrypt via the pod's FIELD_ENCRYPTION_KEY + Prisma client). Code fix: hmp-channel-service commit 98ae91b — IntermediaryService.createConnection now fetches the IntermediaryProviderConfig row by provider code at create time and merges endpointLive/endpointSandbox (selected by credentials.mode) + allowedSourceIps into the credentials JSON before encryption. Operator-supplied values win if present (SUPER_ADMIN can override per-connection). 9 new test cases added. Currently in pipeline #31028 against master; previous pipeline #31018 against the same commit failed on a CI smoke-test flake (see FU-79), so re-triggered. Hotel #2 onboarding can ONLY succeed automatically once this lands in prod.HIGHEng✅ CODE-COMPLETE 2026-05-26 — pipeline #31028 in flight; staging+prod approvals pending.
FU-79CI smoke-test step fails when probe-pod selection picks a Succeeded-phase pod. Surfaced 2026-05-26 on pipeline #31018. Fix shipped via hmp-infrastructure ee7cc01 in ci-cd/azure-pipelines/scripts/smoke-test.sh — heals all 17 backend pipelines simultaneously. Investigation surprise: the obvious field-selector fix (--field-selector=status.phase=Running) was ALREADY shipped 2026-04 in commit 96fc615 (FU-48). #31018's flake was a deeper race: a Running+Ready pod from the OLD ReplicaSet transitioned to Succeeded mid-SIGTERM between kubectl get pod and kubectl exec — the L1 graceful-shutdown window (60s terminationGracePeriodSeconds + preStop sleep 10) widens this race past what the field-selector alone can close. Stronger fix landed: (1) restrict selector to the CURRENT ReplicaSet's pod-template-hash so old-RS draining pods are filtered out entirely; (2) retry candidate-pick + exec up to 3 times on the "completed pod" / "container not found" class of error, non-retryable failures (e.g. 5xx from /api/health/ready) exit immediately. Bash syntax validated. Next pipeline cycle on any backend service exercises it.MediumOp + Eng✅ CODE-COMPLETE 2026-05-26 — hmp-infrastructure commit ee7cc01 to master. Heals all 17 pipelines.
FU-80ResAvenue <AdditionalGuestAmount> Amount="0" on outbound rate XML — downstream OTAs refused rate-parity updates for Lotus Leaf (Property #1). Two-repo silent data-loss path in ARI grid → distribution-fanout chain: property-service fanoutToChannels never queried RoomType.extraAdultRate/extraChildRate or RatePlanRate.extraAdult/extraChild, and channel-service DistributionFanoutService.fanout would have dropped them anyway. Fix: (a) @hmp/service-clients@1.26.1 (hmp-shared-packages b89ce1e) extends DistributionFanoutEditDto with extraAdult? extraChild?; (b) hmp-channel-service ffc6912 extends DistributionFanoutEdit interface + spreads into SyncService.pushRates rates array; (c) hmp-property-service d50450f selects extraAdultRate/extraChildRate from RoomType + new prisma.ratePlanRate.findMany for per-cell overrides, resolves extras = RatePlanRate ?? RoomType and attaches to RATE fanout edits; (d) hmp-web-dashboard 958b586 + dashboard-form-numeric-coercion-followup exposes the form inputs on /settings/room-types. 9 new backend tests (4 channel + 5 property), 4 dashboard tests. Pipelines #31072 + #31073 + #31074 + #31088 succeeded; bot Kustomize commits all 4 envs landed. Operator action still pending: backfill RoomType.extraAdultRate/extraChildRate on Lotus Leaf's 3 room types via the new form, push a fresh rate update, confirm channel_sync_logs.requestXml shows non-zero Amount on both `AgeQualifyingCode="108"` elements, reply to ResAvenue.HIGHEng
FU-81hmp-apps Ingresses recreated by ArgoCD reconcile triggers cert-manager to re-request the LE cert and burned the 5-cert-per-168h LE rate limit (429 urn:ietf:params:acme:error:rateLimited on dashboard.keynova.ai). Currently hmp-web-dashboard-tls + hmp-admin-portal-tls Certificate resources are stuck in Ready=False; the production sites are functional ONLY because the ALB auto-discovered the ACM wildcard cert (*.keynova.ai, two valid copies in ACM). Next allowed LE issuance: 2026-05-26 21:27 UTC (~03:00 IST). Root cause — ArgoCD prune is removing the hmp-web-dashboard-tls / hmp-admin-portal-tls Secret resources from hmp-apps because they aren't in the ApplicationSet's manifest tree (cert-manager creates them out-of-band), so each ingress recreate triggers a fresh ACME order. Two fix paths: (A) Switch hmp-apps ingresses to ACM wildcard cert via alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-south-1:943273444105:certificate/ab7df8be-... annotation + remove the cert-manager.io/cluster-issuer annotation + remove the tls: block. Wildcard cert auto-renews via ACM, no LE dependency. Recommended — eliminates the rate-limit class entirely. (B) Configure ArgoCD Application to exclude cert-manager-managed Secrets from prune via argocd.argoproj.io/sync-options: PruneIgnore=true on each Secret. Surgical but fragile (any future ingress recreate still spawns a Certificate request). Surfaced 2026-05-26 when Lotus Leaf staff couldn't access dashboard.keynova.ai after the FU-80 pipeline rollout. Diagnostic added to feedback_argocd-misses-base-changes-without-overlay-touch family — ArgoCD prune behavior on cross-controller-managed resources is a class of issue worth a follow-up audit.MediumOp + Eng🟡 OPEN 2026-05-26 — prod functional via ACM wildcard fallback; cert-manager Certificates stuck in Ready=False; LE rate limit resets 2026-05-26 21:27 UTC.
FU-82ResAvenue inbound booking parser stored Reservation.totalAmount = SupplierAmount (2890) instead of TotalBookingAmount (3570). Surfaced 2026-05-26 on Lotus Leaf's first prod OTA booking from Booking.com (5304309483). Root cause: ResAvenue's XML carries three overlapping totals on ResGlobalInfo.Total — AmountBeforeTax (net to hotel after commission = SupplierAmount), AmountIncludingMarkup (gross room rate pre-tax), TotalBookingAmount (grand total guest pays incl tax). Parser used RoomStay.Total.@AmountBeforeTax, which equals the gross total ONLY when both commission AND tax are zero. The cert hotel (261) was a 0%-region with no commission — all three fields equal 1002 — so the bug was invisible during cert testing. Lotus Leaf had commission=510 + tax=170, so the divergence appeared on the first real OTA contract. Code fix: hmp-channel-service b64d0f6 — resavenue-xml.parser.ts precedence (1) TotalBookingAmount → (2) AmountIncludingMarkup+tax → (3) legacy AmountBeforeTax. Multi-stay aggregate gross split proportional to per-stay SupplierAmount. 5 new tests; full repo 878/878. Pipeline #31101 in flight. Existing prod row fix: ran hmp-infrastructure/scripts/one-off/fu-82-lotus-leaf-booking-5304309483.sh 2026-05-26 — guarded updateMany corrected booking 5304309483: totalAmount 2890→3570, balanceDue 2890→3570, roomReservation.rateAmount 2890→3400. Idempotent (re-running is no-op).HIGHEng✅ CODE-COMPLETE 2026-05-26 — parser fix in pipeline #31101; prod row fixed via one-off script; awaiting deploy.
FU-83OTA BookingSource enum maps Booking.com → OTHER instead of BOOKING_COM. Surfaced on Lotus Leaf prod booking 5304309483: XML carries <BookingChannel Type="Booking.com"><CompanyName>Booking.com</CompanyName></BookingChannel> but Reservation.source ended up 'OTHER'. Root cause: consumer's VALID_BOOKING_SOURCES set expected canonical enum strings but event.channelCode arrives as the prose CompanyName ("Booking.com"). Cert hotel used "Resavnue Pms" so the prose-name path was never exercised. Fix shipped via hmp-reservation-service 08eaa59 — new exported normaliseBookingSource() helper maps prose names (Booking.com/MakeMyTrip/Goibibo/Expedia/Agoda/Airbnb) + ResAvenue short attribution codes (BKC/MMT/AGD/EXP/GIB) → canonical enum. Canonical values pass through; unknown strings fall back to OTHER (preserves intentional grouping for Google/TripAdvisor/test sources). 24 new spec cases. Baseline 180/180 failures (pre-existing rot) unchanged, +30 new passes. Pipeline #31120 in flight. Existing prod row: booking 5304309483 source='OTHER' can be corrected via a one-off Prisma updateMany mirroring the FU-82 pattern.MediumEng✅ CODE-COMPLETE 2026-05-27 — pipeline #31120.
FU-84<PayAtHotel>Y</PayAtHotel> in OTA XML doesn't propagate to Reservation.paymentStatus = 'PAY_AT_HOTEL'. Surfaced on Lotus Leaf prod booking 5304309483: XML carries BOTH <PayStatus>Unpaid</PayStatus> AND <PayAtHotel>Y</PayAtHotel>, but stored paymentStatus = 'PENDING'. Root cause: derivePayment() PENDING branch in payment-derivation.ts returned PENDING without checking event.payAtHotel. Fix shipped via hmp-reservation-service 08eaa59 — PENDING branch now promotes to PAY_AT_HOTEL when payAtHotel === true; backwards-compatible (when payAtHotel false/absent, stays PENDING). 3 new spec cases. Pipeline #31120 in flight (shipped together with FU-83). Impact post-deploy: dashboard PaymentStatusChip renders amber Pay-at-Hotel indicator; dunning rules will skip pay-at-hotel guests. Existing prod row can be corrected via the same one-off Prisma updateMany as FU-83.MediumEng✅ CODE-COMPLETE 2026-05-27 — pipeline #31120.
FU-85Pre-arrival reservation Folio·Billing tab + Overview folio preview both show empty (no rows, 0 Subtotal/Tax/Balance). Surfaced 2026-05-26 on Lotus Leaf booking 5304309483 immediately after FU-82's row fix — reservation row had correct totalAmount=3570 / taxAmount=170 / rateAmount=3400 but the FolioTable was blank. Not a bug per se — AP-T04-004 (2026-05-23) deliberately removed the client-side folio synthesis so FolioTable only renders server-authoritative rows from frontdesk-service. That closed an audit gap (Flow 4 — client recompute diverged from real folio after splits/discounts/POS charges) but introduced a UX regression: folios are created at check-in by frontdesk, not at booking — so confirmed-but-not-arrived OTA bookings had nothing to render. Fix shipped via hmp-web-dashboard 9055977 — added buildProjectedFolioFromReservation() to folio-view-model.ts that synthesises projected rows from reservation.totalAmount / taxAmount / rooms[].rateAmount / nights when no server folio exists. Each row is labeled PROJECTED in subtext; an amber banner above the table reads "Charges below are projected from the booking total — the actual folio is created at check-in." As soon as the server creates a folio (at check-in, direct charge, or manual posting), hasServerFolio flips true and the view-model switches back to buildFolioViewModel — the AP-T04-004 "no client recompute on an active folio" invariant holds for the entire active lifecycle. Affects both FolioTable instances (Overview center column + Folio·Billing tab). FolioTable.status stays "OPEN" (type only accepts OPEN/CLOSED/VOID); projected state is conveyed via the banner + per-row PROJECTED subtext. Pipeline #31102 in flight. Note: AP-CC-6 SES follow-ups originally claimed FU-85..88; collision resolved 2026-05-26 by shifting them to FU-86..89 to keep this row canonical.MediumEng✅ CODE-COMPLETE 2026-05-26 — pipeline #31102.
FU-86Per-tenant custom-domain SES verification — admin-portal UI + background verifier; EmailService re-wires tenant.customDomain after status=VERIFIED. Surfaced by AP-CC-6 forced-From v1 design.MediumEng + Op🟡 OPEN 2026-05-26 — deferred to v1.1 to keep AP-CC-6 resubmission simple.
FU-87SQS-buffered SES bounce consumer — wrap the HTTPS subscription in SES → SNS → SQS → consumer if sustained bounce rate >1K/day (current target <0.1%).LowEng🟡 OPEN 2026-05-26 — not blocking; preventive upgrade if bounce volume rises.
FU-88DMARC p=none → p=quarantine cutover for _dmarc.mail.keynova.ai after 14 days of clean aggregate reports at dmarc-reports@keynova.ai.MediumEng🟡 OPEN 2026-05-26 — review at T+14d after AP-CC-6 prod apply.
FU-89Apex keynova.ai HTTPS verification (currently the corp-site cert chain) + WAF rate-limit on /api/email/webhook/ses route. ALSO pre-warm SNS X.509 cert cache at module init to eliminate the first-request-per-pod 403 (currently deferred via setImmediate).LowEng🟡 OPEN 2026-05-26 — operational hardening; not blocking.
FU-81hmp-apps Ingresses recreated by ArgoCD reconcile triggers cert-manager to re-request the LE cert and burned the 5-cert-per-168h LE rate limit. Fix shipped via hmp-infrastructure f25c5c0: dropped cert-manager.io/cluster-issuer annotation + tls.secretName from hmp-web-dashboard + hmp-admin-portal base Ingresses. ALB controller falls through to ACM auto-discovery (already working pattern on hmp-api-gateway). Eliminates the LE rate-limit class entirely — wildcard cert renews on ACM's annual schedule. Booking-engine stays on cert-manager DNS-01 (cross-account IRSA, different rate-limit class, unaffected). After ArgoCD syncs, operator should clean up stale cert-manager Certificate CRs + Secrets via kubectl delete certificate hmp-{web-dashboard,admin-portal}-tls --ignore-not-found && kubectl delete secret hmp-{web-dashboard,admin-portal}-tls --ignore-not-found (cert-manager-ingress-shim will auto-delete after the annotation is gone, but explicit delete is faster).MediumOp + Eng✅ CODE-COMPLETE 2026-05-27 — infra master pushed; awaiting ArgoCD sync + operator cleanup.
FU-83-cleanupArchitectural cleanup of FU-83 — prose BookingChannel.CompanyName → canonical ChannelCode normalisation moved from reservation-service consumer to channel-service channel-booking-normalizer. Pre-cleanup, Lotus Leaf's first OTA booking ingested with parsed.channelCode="Booking.com" → ALL_CHANNEL_CODES.has() failed → fell back to sourceProvider='RESAVENUE', making the canonical event's channelCode='RESAVENUE' and the downstream "OTA cancellation via RESAVENUE (RESAVENUE)" reason string identify the intermediary as the channel — incorrect. Fix shipped via hmp-channel-service 3fd195d: new exported normaliseChannelCode(raw, sourceProviderFallback) helper applied in both normalize() (booking) and normalizeCancellation() (cancel). Handles prose names (Booking.com/Goibibo/MakeMyTrip/Expedia/Agoda/Airbnb/Google/TripAdvisor/Amadeus/Sabre) + ResAvenue short codes (BKC/MMT/AGD/EXP/GIB) + canonical enum pass-through. 28 new spec cases (intermediary suite 359/359 pass). The reservation-service consumer's normaliseBookingSource helper is kept as defense-in-depth — its prose-mapping branch is mostly dead code for new events, catches legacy events still on the Kafka topic. Pipeline #31197 in flight.MediumEng✅ CODE-COMPLETE 2026-05-27 — pipeline #31197.
FU-90webhook_events unique key was missing eventType → cancellation/modification audit rows silently dropped. Surfaced on Lotus Leaf prod 2026-05-26T18:15:56Z. Pre-fix the partial-unique (provider, parsedBookingId, status) was meant to dedupe concurrent retries of the same event, but failed to include eventType — and ResAvenue (like most OTAs) reuses the booking's UniqueID on cancellation/modification XML. So the cancellation row's (RESAVENUE, 5304309483, ACCEPTED) clashed with the booking row's (RESAVENUE, 5304309483, ACCEPTED). The Kafka publish ran successfully (reservation got CANCELLED end-to-end via ChannelBookingCancelledConsumer at 18:16:06), but the persistWebhookEvent call afterwards threw P2002, was swallowed by the catch block as a warn, and the audit row vanished. Fix shipped: (a) hmp-channel-service df6f5ad Prisma migration 20260527000000_fu_90_webhook_event_dedup_per_event_status — adds 4-col unique. (b) hmp-channel-service 2110f02 follow-up migration 20260527000001_fu_90_followup_drop_stale_index — drops the stale 3-col unique INDEX that the first migration's DROP CONSTRAINT IF EXISTS silently missed (Prisma's @@unique + map: produces a unique INDEX not a CONSTRAINT — lesson recorded in channel-service CLAUDE.md). (c) AbstractIntermediaryWebhookController.persistWebhookEvent now distinguishes P2002 (expected dedup → info-level structured log) from real persist failures (error-level + stack). (d) Backfill script hmp-infrastructure/scripts/one-off/fu-90-backfill-lotus-leaf-cancellation-audit.sh + fixup script fu-90-fixup-leftover-unique-index.sh — both ran 2026-05-27, cancellation audit row for Lotus Leaf booking 5304309483 is now in place with createdAt pinned to the original event time (2026-05-26T18:15:56.942Z). Impact going forward: every OTA partner that reuses the booking UniqueID on cancel/modify (most do) — cancellations/modifications will now write proper audit rows.HIGHEng✅ DONE 2026-05-27 — pipelines #31169 + #31180; prod backfill verified.
FU-91hmp-channel-service Deployment scaled to 0 replicas on qa and staging — surfaced during FU-90 post-deploy diagnostic 2026-05-27 (fu-platform-post-deploy-validation.sh Section 1 reported (no running channel-service pod in qa — skipping) + same on staging). Test + prod have running pods. Either intentional (cost containment on idle envs) or accidental scale-down. Impact: any OTA traffic test against qa/staging fails silently (no pod to process inbound webhooks). For ongoing FU-78/80/82/83/84/90 regression testing on qa/staging, the deployment needs to be scaled back up. Fix path: kubectl --context <qa|staging-ctx> -n hmp-apps scale deployment hmp-channel-service --replicas=2. Verify with kubectl get deploy hmp-channel-service. If the desired-replicas state is being explicitly clamped in the per-env Kustomize overlay, that's the durable source-of-truth to update (overlay patches HPA minReplicas or Deployment.spec.replicas).LowOp🟡 OPEN 2026-05-27 — non-blocking; uncovered during FU-90 validation pass.
FU-92IntermediaryProviderConfig for RESAVENUE on the test env has allowedSourceIps: [] (empty) while prod carries the 3 real ResAvenue prod IPs (123.108.34.111/112/116). Cosmetic on test (no real OTA traffic) but means future Property #2 onboarding tests run on test don't exercise the FU-78 hydration path the same way as prod (any wizard-created connection on test will inherit an empty allowlist and skip IP filtering entirely). Fix path: SUPER_ADMIN call to PUT /api/providers/RESAVENUE/allowlist with body {"allowedSourceIps":["123.108.34.111","123.108.34.112","123.108.34.116"]} against api.test.keynova.ai. Alternatively, update the seed file hmp-infrastructure/seeds/seed-intermediary-providers.sql so fresh-DB rebuilds get the IPs from the start. Note: same gap probably exists on qa + staging (not verified — diagnostic skipped those envs because channel-service was scaled to 0; see FU-91).LowOp🟡 OPEN 2026-05-27 — non-blocking; uncovered during FU-90 validation pass.
FU-93 / RES-AVAIL-SPEC-10hmp-reservation-service src/availability/availability.service.spec.ts — 10 of 21 specs RED; test-only debt, production code is CORRECT. The implementation shipped correctly in three deliberate 2026-04-15/16 refactors and the spec's mocks were never rewired: (a) holds moved plain-Redis-key → Redis HASH (getClient().hgetall/hset, keys holds:${roomTypeId}/hold:${holdId}) but the spec still stubs mockRedisService.get/.set → getHold returns undefined (commit 99d973b AVAIL-02); (b) releaseHold is now idempotent on a missing hold (resolves void, no throw) but the spec still asserts it throws NotFoundException (commit 1d4afe4 AVAIL-03 — the assertion is the stale half, the behavior is intended); (c) room counting migrated xPrisma → PropertyServiceClient.getRooms() for createHold but the createHold — property scoping specs still stub mockPrismaService.room.count (commit 65c6d7d). Why it shipped green all session: the nestjs-service.yml template runs jest with testsFailOnError: false (Plan 3 D.2 advisory relaxation — npm test … || echo "::warning::" swallows the non-zero exit); these reds were only ever tracked as the blanket plan-3-followup-hmp-reservation-service-test-cleanup opt-out, never itemized — surfaced + characterized 2026-06-04 during #44 day-use verification (git-stash-confirmed the 10 predate #44; #44 ADDED 2 passing day-use specs). FIX = rewire the 10 specs to the Redis-hash + PropertyServiceClient.getRooms contract and flip the releaseHold-throws assertion to expect idempotent resolve; do NOT touch availability.service.ts. Documented in src/availability/CLAUDE.md gotcha 7; Boards Task #6453 (KeyNova\PMS-Core).LowEng✅ DONE 2026-06-04 — 10 specs rewired to current contract (propertyClient seams + auto-parsed hold object + idempotent releaseHold), suite 21/21 green (commit 7b5b9be, pipeline 32071); availability.service.ts untouched; #6453 Closed-Verified. Residual: the service-wide testsFailOnError:false flag-flip is still blocked by 3 OTHER rotted suites (booking-engine, groups ×2; cart/checkout.service.spec FIXED 2026-06-04 via the cart-GST work e32ca31 — added missing PropertyServiceClient provider + rewired getOverflowSuggestions getRooms mocks) → item #6458 (RES-4SUITE-ROT, now 3 suites).

7. Engineering hardening for scale​

For "100s of hotels" scale per 2026-05-18 discussion. All ❌ NOT STARTED. Verified 2026-05-20.

LevelItemWhyEffortStatusWeight
L1Add preStop hook (sleep 10) to 20 base Deployments~95% of rolling-restart 502s eliminatedS✅ DONE 2026-05-2080
L1Bump terminationGracePeriodSeconds to 60sCleaner pod shutdown alignmentS✅ DONE 2026-05-2080
L2Tune ALB deregistration_delay.timeout_seconds=30 (default 300)Faster target removalS✅ DONE 2026-05-20 — 4 ingresses patched60
L2Bump prod HPA minReplicas from 2 to 3 on critical servicesRotation impacts 1/3 not 1/2S✅ DONE 2026-05-20 — 6 critical services60
L3Migrate to RDS IAM auth (Plan 2 already enabled flag; app needs to use it)Eliminates password-rotation impact entirelyM🔴17
L4AWS RDS Proxy (optional, transparent rotation + pool)Cleanest credential isolation; +~$15/moM🔴10
OpSchedule prod RDS rotation for off-peak 02:00 ISTOperator-controlled timingS🔴40

8. Stakeholder sign-offs​

Plan 6 Spec §14 — 5 decisions​

IDQuestionOwnerDeadlineStatus
Q1Reserved Instance commitment timing (hybrid: T+30 baseline + T+90 expand)CFO + SRE Lead2026-07-15🟡
Q2GuardDuty non-prod (keep)CISO + SRE Lead2026-06-01🟡
Q3Slack scheme (per-env channels)SRE + Eng Lead2026-05-30🟡
Q4On-call rotation (3 named approvers — also hard blocker §1)VP Eng + SRE Lead2026-06-15🛑 BLOCKER
Q5Backup retention sign-off (also hard blocker §1)Legal/DPO + CISO + SRE Lead2026-06-15🛑 BLOCKER

CIC sign-offs (Compliance Initiative Charter)​

CICDomainStatus
CIC-1Travel exception WAF🛑 OPEN
CIC-2Rate-limit office IP🛑 OPEN
CIC-3TLS 1.2 ALB🟡 Evidence captured; signatures pending
CIC-4Razorpay rotation + demo guard🛑 OPEN
CIC-5Vault Lock governance🟡 Evidence captured; signatures pending

Sales & Marketing (per 2026-05-16 playbook)​

ItemOwner
VP Sales + CFO sign-off on pricing tiersStakeholder — 🛑 BLOCKED (reconciled 2026-06-04)
CMO + CEO sign-off on marketing planStakeholder — 🛑 BLOCKED (reconciled 2026-06-04)
Board sign-off on KPIs + org/comp planStakeholder — 🛑 BLOCKED (reconciled 2026-06-04)

9. Coverage gates​

From property-1-coverage-2026-05-09.md — Property #1 Gold Standard Charter gates ALL STILL FAIL:

GateThresholdLatest measuredDelta
5 — Backend overall statements≥80%~58%❌ −22 pts
6 — Backend critical-path (auth/payment/frontdesk/checkin/checkout/night-audit/booking-engine)≥90%57-58%❌ −32 pts
8 — Frontend overall (dashboard + booking-engine)≥70%dashboard 57.62% / booking-engine 14.86%❌ −12 to −55 pts

Service-level coverage delta:

  • hmp-auth-service: 58.43% (gate 90%, −32 pts)
  • hmp-payment-service: 57.16% (gate 90%, −33 pts)
  • hmp-web-dashboard: 57.62% (gate 70%, −12 pts)
  • hmp-booking-engine: 14.86% (gate 70%, −55 pts — worst gap)

Critical-path services NOT YET MEASURED: hmp-frontdesk-service, hmp-reservation-service, hmp-property-service, hmp-housekeeping-service (gate 6 needs them all).

AP-T16-002: ✅ DONE 2026-05-23 (1ad407b) — the "309 failures" claim was stale; a fresh run found only 4 (incl. a genuine product bug — WorkOrderCard.ageLabel fed an unguarded date to date-fns → RangeError crashed the Work-Order Kanban board). Dashboard suite now 1592/1592 green (215 files).

AP-T16-004: --coverage runs — UNBLOCKED. The dedicated test-rot cleanup pass landed (10 local branches across 5 services; 187 specs + 3 tsc errors + 1 latent type-hygiene gap fixed; 0 product bugs found; see the 2026-05-23 test-rot change-log entry). After those 10 branches merge, coverage runs become meaningful. (Still env-dependent and out of scope: ~185 reservation-service e2e specs that need live Redis/Postgres in CI — infrastructure ticket, not code rot.)

10. Top 25 prioritized actions (weighted ranking)​

Calculated using (priority × impact) / effort formula. Higher = better ROI.

Status updated 2026-05-21 — 20 of the top 25 are now ✅ closed (FU-46 closed 2026-05-21). The 5 remaining are operator/stakeholder-gated.

Status updated 2026-05-23 — the parallel-agent consolidation batch closed every remaining non-gated engineering item in the top-25 (AP-T08-005 now ✅ DONE). What's left is operator/stakeholder-only: FU-51/C2 SendGrid→SES (AWS re-review), Reloader validation (operator, due 2026-05-24), Op RDS off-peak scheduling, Q4 on-call rotation.

Status updated 2026-05-24 — verify-first sweep closed 9 more items as silent-shipments (AP-T07-006, AP-T10-007, AP-T10-008, AP-T11-004, AP-T13-006, AP-T06-001, AP-T01-006, AP-T03-003, FU-56). See change log entry 2026-05-24 for details.

Status updated 2026-05-24 (afternoon) — FU-52 PROD-VALIDATED end-to-end + FU-65 CLOSED. Overnight cron(0 21 ? * SAT *) fired prod RDS rotation at 03:41:21 IST Sunday; ESO+Reloader restarted all 17 RDS-touching backends within ~34s; cadence reset to AWS-managed weekly. Top-25 row 10 (Reloader validation) flipped to ✅ DONE. Of the 5 remaining top-25 items: 2 are now closed (Reloader validation + AP-T08-005), leaving 3 stakeholder/operator-gated: FU-51 (SendGrid prod key — superseded by SES migration, see OP-2/FU-30), Op RDS rotation scheduling, Q4 on-call rotation.

RankScoreIDTitleOwnerStatus
1100FU-51Provision real SendGrid prod keyOp✅ DONE (reconciled 2026-06-04) — superseded by SES (AP-CC-6); original SendGrid ask obsolete
2100FU-46 / AP-CC-3RS256 JWT keys for staging+prodOp + Eng✅ DONE 2026-05-21 — RS256 in all 4 envs, ALLOW_HS256 removed from 16 overlays, folded into Terraform
3100AP-T05-001Fix rate recalc on extend stayEng✅ DONE — verified shipped 2026-05-20
4100AP-T07-002Wire kitchenGateway.emitKotCreated()Eng✅ DONE — verified shipped 2026-05-20
5100AP-T08-001Replace irn = 'Pending' literal in PDFEng✅ DONE — verified shipped + stale spec fixed 2026-05-20
6100AP-T10-002Single source of truth for dashboard metricsEng✅ DONE — verified shipped 2026-05-20
7100AP-T10-004Mobile dashboardApi URL fixEng✅ DONE — verified shipped 2026-05-20
8100AP-T12-003payment_capture: 1 to Razorpay order bodyEng✅ DONE — verified shipped 2026-05-20
9100AP-CC-1CI check for gateway @All decorator parityEng✅ DONE — verified shipped + 2 P0 drifts fixed 2026-05-20
10100Reloader validationForce-rotate test cluster RDSOp✅ DONE 2026-05-24 — test-cluster walkthrough 2026-05-23 + prod end-to-end overnight rotation 2026-05-24 (FU-52 prod-validated, FU-65 closed)
1180L1 preStopAdd preStop hooks to 20 DeploymentsEng✅ DONE 2026-05-20 (hmp-infrastructure fb69a67)
1280L1 terminationGracePeriodBump to 60sEng✅ DONE 2026-05-20 (same commit)
1360L2 deregistration_delayALB annotation tuneEng✅ DONE 2026-05-20 — 4 base ingresses
1460L2 HPA minReplicasBump prod HPA from 2 to 3Eng✅ DONE 2026-05-20 — 6 critical prod overlays
1550Op rotation schedulingSchedule prod RDS off-peakOp🟡 OPEN — operator AWS console task
1633AP-T05-002Post EXTENSION_NIGHT folio CHARGEEng✅ DONE — verified shipped 2026-05-20
1733AP-T07-001Rewrite KDS Electron appEng✅ DONE — verified shipped 2026-05-20
1833AP-T08-003B2B IRN GuestServiceClient wiringEng✅ DONE — verified shipped 2026-05-20
1933AP-T08-004GuestRetentionTimer + 90d eraseEng✅ DONE — verified shipped 2026-05-20
2033AP-T08-005GuestErasedConsumer in frontdesk + notificationEng✅ DONE 2026-05-23 — frontdesk eca3feb + notification 5f39f4b; all 7 services covered
2133AP-T09-002Analytics dispatcher night.audit.completedEng✅ DONE — verified shipped 2026-05-20
2233AP-T10-003QA empty-MV root causeEng✅ RESOLVED 2026-05-20 — not a bug; see FU-54/FU-55
2333AP-T10-005GSTR-3B prep endpointEng✅ DONE — verified shipped 2026-05-20
2450Q4 on-call rotationSet up 3 named approversVP Eng + SRE Lead🛑 BLOCKED — stakeholder sign-off 2026-06-15
2550FU-53metrics-server install on test+qaEng + Op✅ DONE 2026-05-20 — installed all 4 envs

Next-up — the 3 remaining top-25 items (all operator/stakeholder-gated; no fresh engineering blocked): FU-51 (SendGrid prod key — superseded by SES migration, see OP-2/FU-30), Op RDS rotation scheduling, Q4 on-call rotation.

11. Verification needed (status uncertain)​

Items where my best evidence is git log markers / branch existence — operator should verify these in code before marking done:

IDStatus (verified 2026-05-23)
AP-T01-004✅ DONE — signup/page.tsx + forgot-password/page.tsx both exist + wired in admin-portal
AP-T02-001✅ DONE — latitude/longitude/googleMapsUrl in schema.prisma:115-117 + DTOs:227-243
AP-T03-001✅ DONE — idempotencyKey schema.prisma:125-126 + @@unique([propertyId, idempotencyKey]):159
AP-T03-002✅ RESOLVED — closed by A5 (§12); booking-engine now uses per-tenant publishable keys
AP-T06-001FRRO runbook — check hmp-docs/compliance/ (not re-verified this pass)
AP-T10-001✅ DONE — adr/revpar computed on every upsert (aa90ac9) + backfill script
AP-T11-001✅ DONE — all 4 @All decorators + services.config.ts entries present
AP-T11-006✅ DONE — CHANNEL_MODE: "simulator" in docker-compose.qa-aws.yml:291
Plan 4 Task 20✅ DONE — @hmp/api-contracts@1.12.2, simulation-badge.tsx present
AP-T01-006⚠️ Re-check vs current code — the 2026-04-28 audit text it cites (forgot-password DTO / welcome email / OnboardingController role) is stale; AP-T01-001/002/003 are all ✅ DONE

12. Prod-onboarding fixes​

Prod-onboarding fixes — LOTUS LEAF RESORT (2026-05-22 → 23)​

First paying customer onboarded on prod (LOTUS LEAF RESORT — tenant 0b25fc43-313b-43c8-b908-b02c1bf636fe, property b58a1a50-bfa7-468b-957c-de7a6e28b8d8). 18 issues surfaced across the launch flows; 16 closed. Full detail + commits: docs/superpowers/plans/2026-05-22-prod-onboarding-fixes.md.

ItemWhatStatus
A1–A4Booking-engine domain — host-mapping SUPER_ADMIN scoping, runtime env detection, lotus.book.keynova.ai mapping, *.book.keynova.ai ACM cert✅ DONE
A5Booking-engine publishable API key — per-(tenant,property) key, lazy get-or-create, delivered via resolve-host. Brainstorm→spec→plan→built across 5 repos, reviewed task-by-task + final E2E review. Verified live on prod (resolve-host returns the key; /booking-engine/* → HTTP 200). Closes AP-T03-002.✅ DONE
B1hmp-cross-service-urls ConfigMap → GitOps (_shared kustomize base, 4 envs)✅ DONE (reconciled 2026-06-04) — code shipped; operator ArgoCD adoption
B2INTERNAL_SERVICE_TOKEN → Terraform-minted + ESO + Reloader, RS256 keypair-linked⛔ OPS-GATED — code shipped; operator terraform apply per env
C1Onboarding wizard created tenant-admin emailVerified=false✅ DONE
C2SES production access (supersedes FU-51 SendGrid) — case resubmitted 2026-05-31 with full AP-CC-6 + AP-CC-6-002 remediation (SNS bounce/complaint pipeline, RFC 8058 List-Unsubscribe, functional self-service unsubscribe form, SES suppression-list IAM, IRSA, public AUP/Anti-Spam/Privacy/Unsubscribe pages on keynova.ai CloudFront). See ap-cc-6-ses-prod-access-use-case-2026-05-26.md.🛑 BLOCKED — AWS re-review (~24-48h)
D1Mobile guest check-in stale-read-model guard✅ DONE
D2Property rename✅ DONE
D3POS "Send To Kitchen" Validation Failed — Prisma Decimal string-serialised; DTO needed @Type(() => Number)✅ DONE
—admin-portal /channel-providers crash (ApiResponse envelope not unwrapped, a53c777)✅ DONE
E1/E2ResAvenue prod go-live — provider config + connect LOTUS LEAF RESORT🛑 BLOCKED — runbook ready (hmp-infrastructure/monitoring/runbooks/resavenue-prod-go-live.md); needs per-hotel creds from ResAvenue
F1Gateway rate-limit keying — replaced the spoofable left-most-XFF band-aid with Express trust proxy + verified-JWT 3-tier keying. Brainstorm→spec→plan→built across 3 repos, reviewed task-by-task + final attack-model review.✅ DONE — deploying (runs 30782/83/84)
F2NextAuth refresh-storm (invalid refresh token → tight retry loop)✅ DONE

Open: C2 (external — AWS), E1/E2 (fast-follow — ResAvenue creds). Non-blocking tails: B1/B2 operator apply, F1 prod-gate approvals, the cosmetic run-30778 ArgoCD sync. The PMS itself (front desk, reservations, housekeeping, check-in/out, POS, dashboard, booking site) is fully functional + verified on prod.

Specs/plans: docs/superpowers/specs/2026-05-22-a5-booking-engine-publishable-api-key-design.md

  • …-f1-gateway-rate-limit-keying-design.md (and the matching plans/ files).

13. Property #1 Dashboard Readiness​

Property #1 Dashboard Readiness — PA/PB/PC/PD (2026-05-23 → 24)​

Post-Property-#1-onboarding audit of hmp-web-dashboard IA + self-service surfaces: can a freshly-onboarded Property Admin manage Property details, Rooms, Rates, Taxes, Channels, Payments, etc. from the dashboard alone without contacting KeyNova staff?

Report: docs/superpowers/specs/2026-05-23-property-admin-dashboard-readiness-report.md (25 items: 7 P0 + 9 P1 + 9 P2, plus 8 P3 follow-ups surfaced during code review).

P0 plan: docs/superpowers/plans/2026-05-23-property-admin-dashboard-readiness-p0.md (P1/P2/P3 executed via subagent dispatch directly from the spec — sister-batch precedent of P0 established the patterns).

Branch convention: prop1/PX-readiness-pY-batch (parallel to platform-audit audit/AP-TXX-NNN-…). All work landed on hmp-web-dashboard — single-repo batch, no other repos touched.

Batch summary​

BatchBranchCommitsMerged-to-master HEADStatus
P0 (7 items)prop1/PA-readiness-p0-batch7 (PA-1..PA-7)f25fc32✅ DONE
P1 (9 items via 6 commits)prop1/PB-readiness-p1-batch6 (PB-1..PB-6)00c91a9✅ DONE
P2 (9 items via 4 commits)prop1/PC-readiness-p2-batch4 (PC-1..PC-4)e9f07ea✅ DONE
P3 (5 of 8 follow-ups via 2 commits)prop1/PD-readiness-p3-batch2 (PD-1, PD-2)76d527b✅ DONE

Pipeline runs: #30836 (20260523.5, locked to P2 HEAD e9f07ea) + #30840 (20260524.1, locked to P3 HEAD 76d527b — canonical final gate).

Per-item closure (28 of 33 worked items closed; 5 deferred)​

IDTitleStatus
PA-1Remove dead /settings/notifications tile from Settings hub✅ DONE
PA-2Dedupe duplicate Security/Safeguard-Audit-Trail tiles + surface orphaned /settings/security/mfa✅ DONE
PA-3Retire legacy /settings/notification-templates (308 → /settings/communications/templates)✅ DONE
PA-4Fix /settings/communications/providers/sms hardcoded placeholder-tenant-sms-cred-id + bootstrap providerConfigApi/useProviderConfig for PA-5✅ DONE
PA-5Tenant Email (SendGrid) credential CRUD on dashboard — port from admin-portal /tenants/[id]/providers/email (13 sub-components + API + hooks)✅ DONE
PA-6Canonical Outlets surface at /pos/outlets (KeyNova HeaderZone design + 308 redirect from /settings/outlets)✅ DONE
PA-7Promote /settings/setup-wizard to top of Property group + dismissable homepage banner✅ DONE
PB-1IA cleanups bundle — 4 tile renames + Billing promote + /chain/properties Add-Property fix (was a toast no-op) + DOW Rate Patterns surfaced + sidebar Season Templates → DOW Rate Patterns rename✅ DONE
PB-2Booking-engine theme picker — port from admin-portal /properties/[id]/booking-engine-theme (6 KeyNova presets)✅ DONE
PB-3Tenant template-variables editor — port from admin-portal /tenants/[id]/template-variables (the {{ custom.<key> }} bag)✅ DONE
PB-4SMS + WhatsApp tenant credential CRUD via PA-5's shared <ChannelProviderManager>✅ DONE
PB-5Channel routes + mappings editable — Path A real CRUD via existing upsert endpoints in hmp-channel-service IntermediaryController✅ DONE
PB-6OTA self-onboarding — structured "Request a channel connection" form via /api/support-tickets (replaces mailto:support@keynova.ai empty-state)✅ DONE
PC-1Surface 3 orphaned routes (/settings/in-room-tech, /settings/smart-locks, /settings/currency) in Settings hub + add /settings/audit-trail to sidebar + remove dead comment refs in /notifications/page.tsx:6✅ DONE
PC-2Channel nomenclature standardization — 35 user-facing strings across 17 files. "Channel" for OTA, "Channel Manager" for aggregator. Internal symbols (IntermediaryConnection types, /intermediary/connections paths) left unchanged.✅ DONE
PC-3/settings/payment-providers "Guest Checkout Status" panel — active merchant account, mode (LIVE/SANDBOX/MOCK), provider, scope, View-payment-activity CTA✅ DONE
PC-4Surface /settings/cancellation-policies link from rate-plan form when Refundable is ticked✅ DONE
PD-1P3 test polish bundle — settings-page.test.tsx mock drift fix (mirror of PA-1) + PA-2 regex tightened to exact heading match + PA-6 brittle fs-based test gets a data-driven render + PA-4 ACTIVE-filter createdAt tie-breaker (applied in 2 places)✅ DONE
PD-2Booking-engine theme preview URL runtime config — 3-layer host resolution (tenant host-mapping → env-derived → amber "Preview unavailable" banner). Helper at src/lib/booking-engine-host.ts. Closes the PB-2 follow-up.✅ DONE

Deferred (5 items — not blocking Property #1 launch)​

IDTitleReason
Report P2-21Provider-enrolments dashboard surfacePer readiness report — "blocker only when Tier-2 partner contracts start being written"; not active today
Report P2-25Clarify "Security" tile in Users & SecurityObsolete — PA-2 already renamed the tile to "Security & MFA" pointing at /settings/security/mfa
P3-2Codebase hex-literal → Tailwind utility cleanupLarge standalone refactor (~60+ files); better as its own initiative. PA-7 review confirmed inline-hex IS the de-facto codebase convention (Tailwind iris-*/text-text-* utilities defined but zero consumers in src/)
P3-6PB-5 route isActive flag + mapping DELETE✅ DONE (reconciled 2026-06-04) — Needs backend: UpdateChannelRouteDto not wired in hmp-channel-service controllers; no DELETE /intermediary/mappings/:id exists
P3-8PC-3 last-capture / failure-reason on status panel🟡 OPEN (reconciled 2026-06-04 — backend dependency available) — Needs hmp-payment-service to expose /merchant-accounts/:id/health or /payments/recent?status=...

Open follow-ups discovered during the pass​

None blocking. The LATE_CHECKOUT_HOUR is not exported from @hmp/constants build warning surfaced repeatedly in builds during this pass — pre-existing, not introduced by prop1 work. Logged as a P3-style cleanup for whoever next touches @hmp/constants.

Memory tracker​

memory/project_prop1-readiness-batch-shipped.md (file project_prop1-readiness-pipelines-pending.md — name predates closure) keeps the rolling state for resumption across sessions. Pipeline IDs + per-commit SHAs + the 5 deferred items + their reasons are all recorded there.

✅ CLEARED — verified DONE 2026-06-04 (reconciliation wf_16d0c027)​

180 of 281 tracked items were verified DONE against real code + per-repo CLAUDE.md (source-of-truth rule) by the tracker↔board reconciliation on 2026-06-04. They are collapsed here so the sections above show only what is genuinely pending. Audit-of-record rows (§2 P0 / §3 P1 / §4 P2) remain in place marked ✅ to preserve the 47/41/39 audit counts; this is their cleared index. Still-open after this pass: 50 OPEN · 22 PARTIAL · 26 stakeholder/external-BLOCKED · 2 unverifiable.

Cleared manifest (180 items) — click to expand

Property #1 audit (AP-T##) — P0/P1/P2 (78)​

  • AP-T01-001 — tenantId optional on ResetPasswordRequestDto · auth CLAUDE.md Gotcha 9 + /auth/password/reset table row: tenantId optional, findMany({email}) resolves user; live in auth.service.ts
  • AP-T01-002 — Dispatcher subscribes TENANT_CREATED welcome+verify · notification-service src/dispatcher/tenant-lifecycle.consumer.ts + email-verification.consumer.ts (+specs) render welcome/verify-email on tenant.creat…
  • AP-T01-003 — Allow ADMIN on OnboardingController · auth CLAUDE.md Gotcha 10: OnboardingController now @Roles('SUPER_ADMIN','ADMIN'), ADMIN tenant-scope check; /onboarding row
  • AP-T01-004 — forgot-password + signup pages in admin-portal · admin-portal src/app/(auth)/forgot-password/page.tsx + (auth)/signup/page.tsx both exist with tests/page.test.tsx
  • AP-T01-005 — Public GET /billing/plans · auth-service CLAUDE.md gotcha 12: BillingController.getPlans() carries @Public(); endpoint table line 87 '/billing/plans GET Public'. Tracker line 154…
  • AP-T01-006 — Other Flow 1 P1 items (placeholder/redundant) · Tracker line 155/488: placeholder row, doc-drift; all referenced Flow-1 items (T01-005, T01-007) ✅. Verified 2026-05-24.
  • AP-T01-007 — OnboardingService xPrisma deprecation — delete or migrate · auth-service CLAUDE.md gotcha 11: OnboardingService now calls PropertyServiceClient saga; 'SeedDefaultsService (xPrisma) has been removed'. Commit cd4…
  • AP-T02-001 — lat/long/googleMapsUrl on Property · property schema.prisma:120-122 latitude/longitude/googleMapsUrl + property.dto.ts:227-243 (Google Maps URL validated)
  • AP-T02-002 — S3/MinIO + AWS_S3 env + AssetType · docker-compose.dev-all.yml:259 MinIO sidecar + AWS_S3_BUCKET env (AP-T02-002); AssetType values noted in schema:829; asset.service.ts upload-url
  • AP-T02-003 — Dashboard image-upload widget · ImageUploadWidget wired into settings/property/page.tsx:2957 + room-types/page.tsx:874; 15 specs; dashboard CLAUDE.md L385 section. PARTIAL note is fo…
  • AP-T02-004 — Re-seed QA Property 40b29005 · tracker row98: 7e0b9d5 comprehensive maximalist seed + Plan 4 demo enrichment 2026-05-18..20; auto-seed pipeline
  • AP-T02-005 — Add inclusive Boolean column to TaxConfiguration · property-service CLAUDE.md gotcha 13: TaxConfiguration.inclusive Boolean @default(false); DTOs persist on create+update. Tracker 1cec19a.
  • AP-T02-006 — Class-validator schema for CancellationPolicy.rules[] · property-service CLAUDE.md gotcha 14: cancellation-rules-validator.ts Zod schema (5 chargeType values), validateCancellationRules() called on create+u…
  • AP-T03-001 — idempotencyKey on Reservation + enforce · reservation schema.prisma:134 idempotencyKey + @@unique([propertyId,idempotencyKey]):168; CLAUDE.md Gotcha 13; idempotency-key.decorator.ts; controlle…
  • AP-T03-002 — DB-backed IApiKeyService + rotate seed key · reservation CLAUDE.md Gotcha 17+28(A5): ApiKey table replaces property.settings blob, ApiKeyService wired as API_KEY_SERVICE; api-key.service.ts; no s…
  • AP-T03-003 — Decide theme strategy: KeyNova brand OR document direction-a/b · dashboard CLAUDE.md line 20 + root CLAUDE.md line 106: KeyNova is the chosen default theme via @hmp/design-tokens; Ink&Brass deprecated. Strategy deci…
  • AP-T04-001 — Re-run seed-master.sh QA RDS Property #1 · tracker row101: auto-seed pipeline running on every change (2026-05-19); seed-master.sh canonical
  • AP-T04-002 — Phase 1 mobile reservations decision · _data/decisions/phase1-mobile-scope.yaml status:decided decided_at 2026-04-29; mobile-reservations-browse in deferred[]; decision doc exists. Descope …
  • AP-T04-003 — Wire guestName in search() via GuestServiceClient · reservations.service.ts:3522-3526 filters Guest read-model firstName/lastName case-insensitive OR (no cross-service HTTP). CLAUDE.md AP-T04-003 line 5…
  • AP-T04-004 — Replace client-side FolioTable with rows from res.folios[].transactions[] · dashboard CLAUDE.md line 469-479: FolioTable renders authoritative server folio rows via buildFolioViewModel(folios); client rate×nights synthesis del…
  • AP-T05-001 — Rate recalc on extend stay · reservations.service.ts:2451 AP-T05-001 rate recalc on date change; CLAUDE.md Gotcha 14 (+#4/#35/#36 amendments); extend-stay-gst.spec.ts
  • AP-T05-002 — EXTENSION_NIGHT folio CHARGE for added nights · reservations.service.ts:2516 EXTENSION_NIGHT folio CHARGE; CLAUDE.md Gotcha 15; now atomic via ReservationOutbox in $transaction (RANK 8)
  • AP-T05-003 — Nginx ws proxy QA + prod · infrastructure CLAUDE.md L269-286: nginx /ws/housekeeping/ + /ws/kitchen/ routes; staging/prod/qa-aws compose carry socket.io Upgrade proxy
  • AP-T05-004 — 'Move Room' + 'Extend Stay' CTAs on /reservations/:id · dashboard CLAUDE.md line 454-463: reservation detail gained Move Room + Extend Stay in-place dialogs posting /front-desk/room-move + PUT /reservations…
  • AP-T05-005 — Consolidate RoomOpsService.roomMove + CheckInService.roomMove · frontdesk CLAUDE.md gotcha 321 + src/checkin/CLAUDE.md gotcha 10: RoomOpsService.roomMove is the ONE canonical impl; CheckInService.roomMove is a thin…
  • AP-T05-006 — Post folio CHARGE for paid-upgrade rate-difference inside room-move tx · frontdesk CLAUDE.md gotcha 321: upgradeType=PAID posts ROOM_UPGRADE folio CHARGE inside the room-move $transaction; idempotent dedup key. 8b17553.
  • AP-T06-001 — FRRO SOP runbook + IndianFrroClient interface · hmp-docs/docs/operations/frro-form-c.md SOP runbook + form-c-frro.yaml; @hmp/integrations frro/indian-frro-client.interface.ts + manual-upload-frro-cl…
  • AP-T06-002 — Dedicated GuestVisa model · guest schema.prisma:674 model GuestVisa + CLAUDE.md Gotcha 12; migration 20260523000000_guest_visa_frro_fields; POST/GET/PUT /guests/:id/visas; replac…
  • AP-T06-003 — Kiosk useCheckInFlow OR descope · _data/decisions/phase1-mobile-scope.yaml status:decided; kiosk-self-checkin in deferred[]. Explicit descope decision recorded; tracker 'needs sign-off…
  • AP-T06-004 — Mobile staff frontdesk OR descope · _data/decisions/phase1-mobile-scope.yaml status:decided; mobile-frontdesk-checkin in deferred[]. Descope decision recorded 2026-04-29; tracker 'needs …
  • AP-T06-005 — Switch ID-photo + signature uploads from base64 to /uploads/presigned-url · frontdesk CLAUDE.md gotcha 30 (AP-T06-005): self-checkin base64 idImage/signature removed; presigned PUT URL flow (idDocumentUrl/signatureUrl @IsUrl).…
  • AP-T06-009 — Boot-time check: fail fast if ENCRYPTION_KEY unset in prod · auth-service src/app/app.module.ts:129-142 onModuleInit() throws when NODE_ENV=production && !ENCRYPTION_KEY. Same pattern documented in property-serv…
  • AP-T07-001 — Rewrite KDS Electron app /kitchen ns · kds-kiosk CLAUDE.md Gotchas 1-3: VITE_WS_URL=/kitchen namespace, auth.token JWT, colon-delimited kot:created event names fixed
  • AP-T07-002 — Wire emitKotCreated into generateKOT · pos orders.service.ts:734 kitchenGateway.emitKotCreated in generateKOT; pos CLAUDE.md Gotcha 1; emits kot:created WS + pos.kot.created Kafka
  • AP-T07-003 — Drop /pos/ prefix from mobile F&B · mobile-apps packages/api-client/src/posApi.ts:7 AP-T07-003 no /pos prefix; tests/posApi.test.ts asserts canonical URLs
  • AP-T07-004 — Extend FOLIO_CHARGE_POSTED payload with CGST/SGST/IGST/customerState · pos-service CLAUDE.md gotcha 9: FolioChargePostedPayloadV2 always includes cgst/sgst/igstAmount, customerState, placeOfSupply. Consumer migration 2026…
  • AP-T07-006 — Map transactionCode from orderType (ROOM_SERVICE/CANTEEN/etc) · pos-service CLAUDE.md gotcha 10: mapOrderTypeToTransactionCode() in folio-charge-posted.event.ts; ROOM_SERVICE→POS_ROOM_SERVICE etc; legacy 'FnB' gone…
  • AP-T08-001 — Real IRN+QR replaces 'Pending' literal · frontdesk CLAUDE.md Gotcha 20: Folio.irn/signedQRCode/ackDate/ackNumber columns; InvoicePdfService reads folio.irn; 'Pending' literal removed; migrati…
  • AP-T08-003 — B2B IRN via company (no null company) · frontdesk einvoice.service.ts:211 includes company relation, folio.company?.gstin→B2B IRN; einvoice/CLAUDE.md L48; specs AP-T08-003 in einvoice.servic…
  • AP-T08-004 — GuestRetentionTimer +90d cron erase · guest schema.prisma:705 GuestRetentionTimer (expiresAt=checkout+90d); CLAUDE.md Gotcha 11: RetentionCronService cron 02:10 erases, CheckoutConsumerSer…
  • AP-T08-005 — GuestErasedConsumer in 7 services · GuestErasedConsumer present in all 7 services: frontdesk(Gotcha30)/pos(L174)/housekeeping(L203)/analytics(L411)/crm(L228)/payment(L129)/notification(L…
  • AP-T08-006 — Promote irn/signedQRCode/ackDate columns onto Folio · frontdesk CLAUDE.md gotcha 20: Folio.irn/signedQRCode/ackDate/ackNumber columns (migration 20260429120000); EInvoiceJobProcessor denorms on COMPLETED;…
  • AP-T08-007 — Route credit-note IRN through EInvoiceJob queue with retry/DLQ · frontdesk CLAUDE.md gotcha 29 + src/einvoice/CLAUDE.md: generateCreditNote queue-routed via EInvoiceJob documentType=CRN, retry+DLQ, idempotent via fi…
  • AP-T09-001 — NA rollback includes OVERSTAY_NIGHT+NO_SHOW · frontdesk CLAUDE.md Gotcha 19: rollbackNightAudit voids transactionCode in ['ROOM','OVERSTAY_NIGHT','NO_SHOW'] in both discovery + bulk void
  • AP-T09-002 — Analytics dispatcher to night.audit.completed · analytics CLAUDE.md L195/412/429: NightAuditConsumer subscribes night.audit.completed, SET-upserts MvDailyOccupancy+MvDailyKpi(MvAdrRevpar)+NightAudit…
  • AP-T09-003 — Implement night-audit-summary PDF/CSV generator · frontdesk CLAUDE.md gotcha 21 + src/night-audit/CLAUDE.md gotcha 8: NightAuditPdfService.generatePdf+generateCsv; GET /night-audit/record/:id/summary?…
  • AP-T09-004 — Add typed currentBusinessDate Date? column on Property · property-service CLAUDE.md gotcha 12 + model row: Property.currentBusinessDate DateTime?; PATCH /properties/:id/business-date gated NIGHT_AUDIT_SERVIC…
  • AP-T09-005 — Promote NA scheduler to k8s CronJob with miss-detection alert · hmp-infrastructure/kubernetes/jobs/night-audit-cronjob.yaml (AP-T09-005, schedule 30 20 * * *, Forbid concurrency); miss-detection alerts HMPNightAudi…
  • AP-T10-001 — MvAdrRevpar adr/revpar on upsert + backfill · analytics reservation.consumer.ts:210 AP-T10-001 computes MvDailyKpi adr/revpar on every upsert; reservation.consumer.daily-kpi.spec.ts; backfill cron…
  • AP-T10-002 — DashboardController.getMetrics queries MvAdrRevpar · analytics CLAUDE.md L93: /analytics/dashboard/metrics reads ADR/RevPAR/todayRevenue from MvAdrRevpar single source of truth (AP-T10-002)
  • AP-T10-003 — Investigate QA empty-MV root cause · tracker row122 RESOLVED 2026-05-20: not a bug — 9/31 MVs populated, consumers healthy, EventStore-empty by design; split into FU-54/FU-55
  • AP-T10-004 — Mobile dashboardApi.getMetrics URL fix · mobile-apps packages/api-client/src/dashboardApi.ts:18 getMetrics GET /analytics/dashboard/metrics (AP-T10-004)
  • AP-T10-005 — GSTR-3B prep endpoint /gst · analytics CLAUDE.md L131-134: GET /reports/gstr3b under GstrModule; MvGstr3bSnapshot populated by GstrService.computeGstr3b; B2B/B2C + CGST/SGST/IGST …
  • AP-T10-006 — Subscribe reservation.modified, fix folio.charge.posted->folio.posted, add folio.closed/kot.completed · analytics CLAUDE.md line 434/436: folio.charge.posted→folio.posted renamed, folio.closed added, ReservationConsumer subscribes reservation.modified, P…
  • AP-T10-007 — Replace /reports/[category] fallback list with backend catalog · dashboard CLAUDE.md line 101: legacy [category] pages deleted 2026-05-01, superseded by v2 catalog driven by GET /api/reports/v2/catalog. Tracker c9b0…
  • AP-T10-008 — Replace admin /analytics demo data with real API · analytics CLAUDE.md line 431 (AP-T10-008): AdminOverviewService.getPlatformOverview cross-tenant; admin portal /analytics deleted fixtures, calls useA…
  • AP-T10-009 — Fix MvDailyOccupancy.totalRooms upsert from PropertyServiceClient · analytics CLAUDE.md line 435 (AP-T10-009): ReservationConsumer sources totalRooms from PropertyServiceClient.getActiveRoomCount; self-heals zeroed row…
  • AP-T11-001 — @All channel/webhook gateway decorators · gateway services.config.ts:589-607 prefixes + proxy.controller.ts:1146 @All('channel-setup')/('channel-stats')/('intermediary')/('webhook-management')…
  • AP-T11-002 — ResAvenue connection row Property #1 · tracker row126: ed88cfe ResAvenue simulator connection seed + PROD cert 2026-05-16; MEMORY confirms Lotus Leaf live on ResAvenue 2026-05-26
  • AP-T11-003 — Register outbound webhook URL ResAvenue · tracker row127: 337b1a0 webhook env vars + seed SQL; channel CLAUDE.md L29 IntermediaryConnection webhookCallbackUrl/webhookSecret (AP-T11-003)
  • AP-T11-004 — Implement MinLOS/MaxLOS push (OTA_HotelBookingRuleNotifRQ) · channel-service CLAUDE.md line 222 (AP-T11-004): MinStay/MaxStay emitted in rate XML per ResAvenue's 2026-05-04 directive; LengthsOfStay removed by pa…
  • AP-T11-005 — Add flow-tests/flow-11-channel-manager.spec.ts end-to-end · flow-tests/src/flows/channel-manager-multi.flow.spec.ts has dedicated 'Scenario AP-T11-005' it('AP-T11-005: pushes MinStay/MaxStay restrictions via ra…
  • AP-T11-006 — Set CHANNEL_MODE=simulator in docker-compose.qa-aws.yml · hmp-infrastructure/docker/docker-compose.qa-aws.yml:291 CHANNEL_MODE: "simulator". channel-service CLAUDE.md line 179 confirms (AP-T11-006). Tracker…
  • AP-T11-007 — channelCodeMap + whitelistedIps for ResAvenue sandbox seed · channel-service CLAUDE.md line 238 (AP-T11-007): credentials.whitelistedIps [203.109.97.241,122.182.6.210] + channelCodeMap reverse-maps 47 ResAvenue …
  • AP-T11-016 — Spec + plan reconciliation · hmp-docs/superpowers/plans/2026-05-01-ap-t11-016-canonical-channel-booking-event.md exists (canonical channel booking event); AP-T11-023 canonical OTA…
  • AP-T11-023 — ResAvenue cert cycle flow test · channel-service CLAUDE.md line 244 (AP-T11-023): ChannelBookingNormalizer.normalizeCancellation → canonical OtaBookingCancelledEvent; applyModificatio…
  • AP-T11-024 — Seed default IntermediaryProviderConfig rows · channel-service CLAUDE.md line 310 (AP-T11-024 Phase 3): seed hmp-infrastructure/seeds/seed-intermediary-providers.sql wired into seed-master.sh creat…
  • AP-T12-001 — Seed MerchantAccount tenant 1 Razorpay sandbox · tracker row128: 1f6db25 Razorpay sandbox MerchantAccount API seed; infrastructure/scripts/seed-razorpay-sandbox-merchant.sh present
  • AP-T12-003 — Add payment_capture: 1 to Razorpay order body · payment-service CLAUDE.md AP-T12-003 (line 257-261): 'payment_capture: 1 is mandatory in the Orders body'; auto-capture confirmed.
  • AP-T12-004 — Build WebhookReconcileService cron (every 15 min) · payment-service CLAUDE.md AP-T12-004 (lines 137-138, 312-358): two-layer reconcile — WebhookReconcileService @Cron 10min + PaymentReconcileService @Cr…
  • AP-T12-005 — Override HARDWARE_MOCK=false in docker-compose.prod-aws.yml · hmp-infrastructure/docker/docker-compose.prod-aws.yml:44 HARDWARE_MOCK: "false".
  • AP-T13-001 — Tenant-scope assert on getCalendar/createQuickReservation · frontdesk calendar.service.ts:100+444 assertPropertyTenant first in getCalendar+createQuickReservation; CLAUDE.md Gotcha 18; calendar.service.spec.ts:…
  • AP-T13-002 — FilterChipsRow query params to backend filter · dashboard CLAUDE.md L35/501/505: FilterChipsRow URL-linked → query params → backend GET /front-desk/calendar?roomTypeId; CalendarService.getCalendar(f…
  • AP-T13-003 — Tape-chart RoomBlocks Block range frontend · dashboard CLAUDE.md L503: BlockRangeDialog wired end-to-end, POST /rooms/:id/block + /rooms/block-batch via roomBlocksApi; 15 specs block-range-dialog…
  • AP-T13-003-multiroom — Multi-room reservation visual continuity · dashboard tape-chart-v2/group-connector.tsx (header 'AP-T13-007: multi-room reservation visual continuity') wired into chart-grid.tsx:404 <GroupConnec…
  • AP-T13-004 — Add mark-no-show to BAR_ACTIONS · dashboard CLAUDE.md line 33 BAR_ACTIONS includes mark-no-show (AP-T13-004); gotcha 2 line 507: visible only for CONFIRMED + arrival≤today, calls reser…
  • AP-T13-006 — Date-range block UI (RoomBlock model or HK MaintenanceRequest) · dashboard CLAUDE.md line 503 (AP-T13-003): BlockRangeDialog posts POST /api/rooms/:roomId/block (single) or /rooms/block-batch (≥2) via roomBlocksApi;…
  • AP-T16-002 · Dashboard suite 1592/1592 green per tracker §9 (commit 1ad407b); 257 dashboard .test files present. 309-failures claim was stale.

Property #1 Dashboard Readiness (PA/PB/PC/PD/P3) (20)​

  • P3-6 — PB-5 route isActive flag + mapping DELETE (backend-blocked) · Backend unblocked: channel CLAUDE.md PATCH /intermediary/connections/:id (isActive, line 114) + DELETE /intermediary/mappings/:id (line 125, 2026-05-2…
  • PA-1 — Remove dead /settings/notifications tile from Settings hub · dashboard CLAUDE.md 'Dead-link triage (PA-1, 2026-05-23)' line 1263; settings-hub.test.tsx coverage.
  • PA-2 — Dedupe Security/Safeguard-Audit-Trail tiles + surface orphaned /settings/security/mfa · dashboard CLAUDE.md 'Audit-trail dedupe + MFA surface (PA-2)' line 1267: Security tile repointed to /settings/security/mfa, renamed 'Security & MFA'.
  • PA-3 — Retire legacy /settings/notification-templates (308 -> communications/templates) · dashboard CLAUDE.md 'Legacy notification-templates retired (PA-3)' line 1275.
  • PA-4 — Fix /settings/communications/providers/sms hardcoded placeholder + bootstrap providerConfigApi · dashboard CLAUDE.md 'SMS providers page uses real cred ID (PA-4)' line 1283; sms/tests/page.test.tsx.
  • PA-5 — Tenant Email (SendGrid) credential CRUD on dashboard — port from admin-portal · dashboard CLAUDE.md 'Email credential CRUD on dashboard (PA-5)' line 1293; settings-hub.test.tsx PA-5 tile spec.
  • PA-6 — Canonical Outlets surface at /pos/outlets + 308 redirect · dashboard CLAUDE.md 'Outlets canonical surface (PA-6)' line 1279.
  • PA-7 — Promote /settings/setup-wizard + dismissable homepage banner · dashboard CLAUDE.md 'Setup Wizard promotion + homepage banner (PA-7)' line 1271.
  • PB-1 — IA cleanups bundle — tile renames + Billing promote + Add-Property fix + DOW Rate Patterns · dashboard CLAUDE.md 'IA cleanups (PB-1)' line 1318; settings-hub.test.tsx PB-1 specs.
  • PB-2 — Booking-engine theme picker — port from admin-portal (6 KeyNova presets) · dashboard CLAUDE.md 'Booking Engine Theme picker (PB-2)' line 1330; settings/branding/theme/tests/page.test.tsx (3 specs).
  • PB-3 — Tenant template-variables editor — port from admin-portal · dashboard CLAUDE.md 'Template Variables editor (PB-3)' line 1342; settings-hub.test.tsx PB-3 tile spec; ADMIN-gated via useRoleGate.
  • PB-4 — SMS + WhatsApp tenant credential CRUD via shared ChannelProviderManager · dashboard CLAUDE.md 'SMS + WhatsApp tenant credential CRUD (PB-4)' line 1386; sms/tests/page.test.tsx + settings-hub.test.tsx PB-4.
  • PB-5 — Channel routes + mappings editable — real CRUD via IntermediaryController upserts · dashboard CLAUDE.md 'Channel routes + mappings editable (PB-5)' line 1362; backed by channel-service intermediary PATCH/DELETE routes (lines 114-125).
  • PB-6 — OTA self-onboarding — Request a channel connection form via /api/support-tickets · dashboard CLAUDE.md 'OTA self-onboarding request form (PB-6)' line 1403: RequestChannelConnectionDialog submits FEATURE_REQUEST to /api/support-ticket…
  • PC-1 — Surface orphaned routes (in-room-tech/smart-locks/currency) + audit-trail in sidebar · dashboard CLAUDE.md 'P2 quick fixes (PC-1)' line 1411; settings-hub.test.tsx PC-1 (4 specs).
  • PC-2 — Channel nomenclature standardization — 35 strings across 17 files · dashboard CLAUDE.md 'Channel nomenclature standardized (PC-2)' line 1443; settings-hub.test.tsx PC-2 asserts no Intermediary/Channel-Provider leaks.
  • PC-3 — /settings/payment-providers Guest Checkout Status panel · dashboard CLAUDE.md 'Payment Providers checkout status panel (PC-3)' line 1426: CheckoutStatusPanel.tsx + tests (5 specs). 4 fields shipped; last-…
  • PC-4 — Surface /settings/cancellation-policies link from rate-plan form when Refundable · dashboard CLAUDE.md 'Cancellation policies surfaced from rate-plan form (PC-4)' line 1463.
  • PD-1 — P3 test polish bundle — mock drift fix + regex tightening + data-driven render · dashboard CLAUDE.md 'P3 test polish (PD-1)' line 1453: useTourGuide mock realign + PA-2 MFA assertion tightened.
  • PD-2 — Booking-engine theme preview URL runtime config (3-layer host resolution) · dashboard CLAUDE.md 'Booking-engine preview URL runtime config (PD-2)' line 1469: lib/booking-engine-host.ts + use-booking-engine-host.ts; no build-ti…

Follow-ups (FU) (40)​

  • FU-1 · trufflehog-scan.sh + .trufflehogignore + template wiring present; infra CLAUDE.md L950 '✅ FU-1 Trufflehog CI (Plan 3 T30)'
  • FU-10 · infra CLAUDE.md L951 '✅ FU-10 pod-security duplicate Namespace (Plan 3 T0)'
  • FU-15 · infra CLAUDE.md L808/L954 'FU-15 RESOLVED; kubernetes_job_v1.logical_dbs in modules/rds-postgres/logical-dbs.tf'
  • FU-16 · infra CLAUDE.md L809 'FU-16 RESOLVED (Loki PVC pinned at 20Gi)'
  • FU-17 · infra CLAUDE.md L810 'FU-17 RESOLVED (Loki helm timeout 1200s)'
  • FU-22 · hmp-infrastructure/.gitattributes L15 '*.tf text eol=lf' (covers .tf for heredoc CRLF leak)
  • FU-23 · terraform/global/azure-devops-mgmt-oidc/ stack exists (iam-role.tf, oidc-provider.tf); infra CLAUDE.md L868 FU-23 closure 2026-05-11
  • FU-29 · infra git: edb4b68/baa7a34 per-service overlays; T27 (3269cee 17 services/env) + T28 archive; Plan 3 overlay subdirs landed
  • FU-32 · infra git eed0ade/e373813 'remove trigger/pr from 7 templates (FU-32 templates portion)' merged
  • FU-4 · infra CLAUDE.md L713 'FU-4 region-lock IAM exemption (resolved)'
  • FU-44 · infra CLAUDE.md L973 — @hmp/cache@1.2.0 + @hmp/messaging@1.6.0 read REDIS_TLS=true; 16-service propagation
  • FU-46 · RS256 in all 4 envs: tls_private_key.jwt in terraform/envs/{test,qa,staging,prod}/service-runtime-secrets.tf; ALLOW_HS256 removed from ALL overlays (g…
  • FU-46-open · RS256 keypair provisioned all 4 envs (terraform/envs/{prod,staging}/service-runtime-secrets.tf tls_private_key.jwt); MEMORY: FU-46 RS256 closed all AW…
  • FU-47 · infra CLAUDE.md L975 — @hmp/observability@1.2.0 rewrites optional injected params; HealthService DI fixed
  • FU-50 · infra CLAUDE.md L944 'FU-50 ✅ RESOLVED 2026-05-17; promote-image.sh 3-attempt rebase-retry loop, self-tested'
  • FU-51-open · infra CLAUDE.md L380/L930 'FU-51 SUPERSEDED 2026-05-21 by SendGrid→AWS SES migration; there is no SendGrid prod key'. MEMORY: AP-CC-6 SES prod-access …
  • FU-52 · infra CLAUDE.md L330 B2 Reloader auto-restart; MEMORY: FU-52 prod-validated end-to-end overnight RDS auto-rotation fired clean 2026-05-24
  • FU-53 · MEMORY: FU-53 DONE 2026-05-20 — operator installed metrics-server v3.13.0 via helm on all 4 envs; kubectl top nodes works everywhere
  • FU-55 · reservation 866391d + frontdesk 208e25d both halves landed; reservation CLAUDE.md L383 FU-55 segmentation fields; frontdesk dual-emit FOLIO_POSTED/CLO…
  • FU-61 · api-gateway commit 1e5fa5c 'audit(FU-61): import MetricsModule... Closes FU-61'; verified app.module.ts:39 MetricsModule imported. Tracker stale.
  • FU-65 · MEMORY 2026-05-24: 'FU-65 = CLOSED' — Sunday-morning verification of overnight Reloader prod auto-rotation done, schedule reset to AWS-managed weekly
  • FU-67 · dashboard 494673a 'lower hardcoded pagination limits to 100 (FU-67 audit)'; MEMORY: 5 bugs fixed, admin/booking/mobile clean
  • FU-7 · infra CLAUDE.md L689/L715/L960 'FU-7 closed 2026-05-11 via forEach+deny+AnyIn rewrite; cluster-wide enforcement'
  • FU-75-active-conn · channel 376e7d0 'hydrate health fields on findAll for dashboard cards' supports Active Connections tab UX; connection update/deactivate cascade in FU-…
  • FU-76-csp · dashboard 79471b6 'CSP allow direct S3 uploads' + f044853 'derive real origin from x-forwarded-host not request.nextUrl' (pod-internal authority leak …
  • FU-76-wizard-cascade · channel CLAUDE.md L354 + commit e2fb994 — POST /intermediary/connections cascade-creates ChannelConnection + ChannelRoute in one tx; 7 tests
  • FU-77-presigned · property-service 8cc9848 'return presigned GET URLs from listImages/registerImage'; dashboard CLAUDE.md L1548 reads presignedDocumentUrl
  • FU-77-prisma-drift · MEMORY: FU-77 drift sweep 60 sites/7 services landed (spa/reservation/guest/frontdesk/channel/notification/property commits); channel 3c9d67c, propert…
  • FU-78-wizard-hydrate · channel 98ae91b 'hydrate endpoint + whitelistedIps from provider catalog at connection create'
  • FU-79 · infra ee7cc01 'smoke-test pod selector restricts to current ReplicaSet + retries on Succeeded race (FU-79)'; heals all 17 backend pipelines
  • FU-8 · infra CLAUDE.md L804 'FU-8 RESOLVED (vpc-cni prefix delegation Plan 2 T12)'; L783 ENABLE_PREFIX_DELEGATION
  • FU-80 · channel CLAUDE.md L352 + commit ffc6912 — extraAdult/extraChild threaded through fanout to pushRates; property-service mirror; 9 tests. NOTE operator …
  • FU-81-le-ratelimit · infra CLAUDE.md L168/L189 FU-81 — hmp-apps Ingresses switched to ACM auto-discovery, drop cert-manager.io/cluster-issuer annotation (no LE rate-limit …
  • FU-82 · channel CLAUDE.md L350 + commit b64d0f6 — parser uses ResGlobalInfo.Total.@TotalBookingAmount (gross), not SupplierAmount; 5 tests
  • FU-83 · reservation 08eaa59 'OTA source mapping + PayAtHotel precedence (FU-83+FU-84)'; ChannelCode→BookingSource BOOKING_COM mapping
  • FU-83-cleanup · channel 3fd195d 'move prose-name→ChannelCode normalisation into channel-booking-normalizer (FU-83 cleanup)'
  • FU-84 · reservation CLAUDE.md L232/L328 — ChannelBookingReceivedConsumer maps event.payAtHotel→paymentStatus='PAY_AT_HOTEL'; commit 08eaa59
  • FU-85 · dashboard 9055977 'projected folio fallback for pre-arrival reservations' + 55f4a73 follow-up off-by-tax correction
  • FU-9 · infra CLAUDE.md L805 'FU-9 RESOLVED (exec-auth in Plan 2 T0)'
  • FU-90 · channel CLAUDE.md L348 + df6f5ad — webhook_events unique key now 4-col (provider,parsedBookingId,eventType,status); follow-up 2110f02 drops stale inde…

Plans 4–7 (11)​

  • Plan4-Task17 · In-cluster seed-runner Job present: kubernetes/jobs/seed-runner/ (job-keynova-seed.yaml, run-keynova-seeds.sh, prod-demo/job-prod-demo-seed.yaml) + ci…
  • Plan4-Task19 · Read-model fanout: analytics-service tenant-read-model.consumer.ts + event-dispatcher.service.ts; @hmp/read-views package (packages/read-views/src/lib…
  • Plan4-Task20 · @hmp/api-contracts@1.14.0 (≥1.10.0) per package.json; dashboard simulation-badge.tsx + tests/simulation-badge.test.tsx present (yellow [DEMO] chip…
  • Plan4-Tasks21-23 · e2e seed smoke handled by auto-seed pipeline (seed-runner-apply.yml + run-keynova-seeds.sh); tracker §5 confirms test/qa/staging covered.
  • Plan4-bis · Asset-mgmt demo enrichment seed present: seeds/14-asset-demo-enrichment.sql + asset-sample-parts.sql + demo-data-generator/src/generators/07-asset.ts.…
  • Plan5-ingresses · api.keynova.ai + live.keynova.ai referenced across prod overlays; dedicated ingress-live.yaml at kubernetes/overlays/prod/hmp-api-gateway/. Live on pr…
  • Plan5-resavenue-first-hotel · First hotel (Lotus Leaf, Property #1) went LIVE on ResAvenue 2026-05-26 with per-hotel creds (channel-service CLAUDE.md:352; memory project_2026-05-26…
  • Plan5-resavenue-prod-handler · ResAvenue prod handler shipped: hmp-channel-service/src/intermediary/providers/resavenue/ (webhook.controller, xml.parser, xml.builder, adapter). Lotu…
  • Plan6-baseline-once · Baseline executed 2026-05-17: compliance/baselines/prod-launch-baseline.yaml + evidence-pack/baseline/B-01..B-20 .txt files all timestamped 2026-05-17…
  • Plan7-Phase1 · Phase 1 inventory + 2-of-N approval authored per tracker §5. Plan 7 plan docs exist.
  • Plan7-Phase2 · Phase 2 data-snapshot skipped per operator decision (seed-only data) — tracker §5. Decision-closed.

Corporate/cross-cutting (AP-CC) (4)​

  • AP-CC-1 — CI check gateway prefix parity · gateway scripts/validate-gateway-routing.ts + .spec.ts + proxy.controller.routing.spec.ts enforce services.config↔@All parity; tracker row485 verified
  • AP-CC-2 — Reseed QA RDS verify data · tracker row134: auto-seed pipeline reseeds QA RDS via seed-master.sh; smoke test passes
  • AP-CC-3 — RS256 migration for WebSocket gateways · pos CLAUDE.md Gotcha 8 + housekeeping CLAUDE.md Gotcha 7: both gateways onModuleInit throw in prod if JWT_PUBLIC_KEY missing, reject non-RS256 alg
  • AP-CC-4 — Remove prod mocks + add placeholders · compose.prod-aws.yml:44-45 HARDWARE_MOCK=false + EINVOICE_MOCK=false; NIC/SENDGRID/RESAVENUE placeholders L180-262; check-no-prod-mocks.sh CI gate

Infra/baseline (B#) (2)​

  • B1 — hmp-cross-service-urls ConfigMap -> GitOps (_shared kustomize base, 4 envs) · infra CLAUDE.md gotcha B1 + base/apps/_shared/cross-service-urls-configmap.yaml + base/apps/_shared/kustomization.yaml + overlays/{test,qa,staging,pro…
  • B2 — INTERNAL_SERVICE_TOKEN -> Terraform-minted + ESO + Reloader, RS256 keypair-linked · terraform/modules/internal-service-token/{main,outputs,variables}.tf + sign-internal-token.js; base/apps/_shared/internal-service-token-externalsecret…

Engineering hardening (L1–L4) (4)​

  • L1-preStop · preStop hook present on 20 backend deployments + 4 frontends (grep count=24) in kubernetes/base/apps/*/deployment.yaml.
  • L1-terminationGrace · terminationGracePeriodSeconds: 60 present on 20 base deployments (grep uniq count=20).
  • L2-deregistration · alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30 on 4 ingresses (api-gateway/dashboard/admin/booking).
  • L2-hpa-minreplicas · JSON patch /spec/minReplicas→3 in 6 critical prod overlays: reservation, frontdesk, auth, payment, api-gateway, property (kustomization.yaml L2 2026-0…

Other (21)​

  • A1-A4 — Booking-engine domain — host-mapping scoping, runtime env, lotus.book mapping, ACM cert · property-service CLAUDE.md gotcha 17/A5 HostMappingsModule + fix/a1b-host-mappings-superadmin-tenant (15 specs); infra A4 ACM gotcha + runbook a4-book…
  • A5 — Booking-engine publishable API key — per-(tenant,property) lazy key via resolve-host (closes AP-T03-002) · reservation-service CLAUDE.md gotcha 28 + PublishableApiKeyController + getOrCreatePublishableApiKey (idempotent); property-service resolve-host retur…
  • C1 — Onboarding wizard created tenant-admin emailVerified=false · auth-service src/tenants/admin-preverified.spec.ts: C1-1 admin-create emailVerified=true, C1-5 self-service emailVerified=false. Tested both paths.
  • D1 — Mobile guest check-in stale-read-model guard · frontdesk CLAUDE.md gotcha 28 (QA-CHECKIN/D1): both checkIn() + GuestCheckInService.confirmCheckIn() delegate to common/reservation-status-reconciler.…
  • D2 — Property rename · property-service PUT /properties/:id (MANAGER+, line 57) updates Property.name; tenant ownership via assertPropertyTenant. Generic property update cov…
  • D3 — POS 'Send To Kitchen' Validation Failed — Decimal string-serialised, DTO @Type(()=>Number) · pos-service CLAUDE.md gotcha 18: @Type(()=>Number) on OrderItemDto.unitPrice/quantity + AddItemDto.quantity; tests src/orders/dto/order.dto.spec.ts (1…
  • F1 — Gateway rate-limit keying — trust proxy + verified-JWT 3-tier keying · api-gateway CLAUDE.md gotcha 10 (F1 2026-05-23): 3-tier keying (verified JWT / login-IP / anon), verifyJwt util, trust proxy=[loopback,uniquelocal] re…
  • F2 — NextAuth refresh-storm (invalid refresh token -> tight retry loop) · dashboard CLAUDE.md 'Access-token refresh — terminal-failure rule (F2, 2026-05-22)' line 826/844: terminal-failure marker prevents the 401-refresh tig…
  • Reloader-prod-validation · reloader.stakater.com/auto annotation on 20 backend deployments + _shared externalsecret (grep count=21). FU-52 prod-validated overnight rotation 2026…
  • Report-P2-25 — Clarify 'Security' tile in Users & Security (obsolete — done via PA-2) · Resolved by PA-2: dashboard CLAUDE.md line 1269 repointed/renamed the duplicate 'Security' tile to 'Security & MFA' → /settings/security/mfa.
  • VERIF-AP-T01-004 — Verify: signup + forgot-password pages exist + wired in admin-portal · hmp-admin-portal/src/app/(auth)/signup/page.tsx (7KB, 22 form/api refs) + forgot-password/page.tsx (3KB, wired). Both present with onSubmit/useMutatio…
  • VERIF-AP-T01-006 — Verify: re-check AP-T01-006 vs current code (stale audit text) · AP-T01-006 is a placeholder/redundant row — not defined in audit Flow-1 section (only T01-005,T01-007 exist at audit L349-350). Tracker L155+436 resol…
  • VERIF-AP-T02-001 — Verify: latitude/longitude/googleMapsUrl in schema + DTOs · hmp-property-service prisma/schema.prisma:120-122 (Float? lat/long + googleMapsUrl w/ AP-T02-001 comment); dto/property.dto.ts:227,232,243 + 416,432 w…
  • VERIF-AP-T03-001 — Verify: idempotencyKey schema + @@unique · reservation-service schema.prisma:134 idempotencyKey + :168 @@unique([propertyId,idempotencyKey]); payment-service schema.prisma:209 + :226 @@unique([…
  • VERIF-AP-T03-002 — Verify: booking-engine uses per-tenant publishable keys (closed by A5) · booking-engine resolves publishableApiKey via resolve-host (useHostBootstrap) + uses providerPublicKey from backend (useAmendDifferentialCheckout.ts:5…
  • VERIF-AP-T06-001 — Verify: FRRO runbook in hmp-docs/compliance/ · Runbook present: hmp-docs/docs/operations/frro-form-c.md (4.7KB staff manual-upload runbook) + _data/compliance/form-c-frro.yaml registry, commit 8d0f…
  • VERIF-AP-T10-001 — Verify: adr/revpar computed on every upsert + backfill script · analytics reservation.consumer.ts:245-263 computes adr+revpar on every upsert (per-night ADR guard); src/scripts/backfill-mv-daily-kpi.ts header 'AP-T…
  • VERIF-AP-T11-001 — Verify: all 4 @All decorators + services.config.ts entries present · gateway proxy.controller.ts:1146-1203 has @All for channel-setup/channel-stats/intermediary/webhook-management (+ /*); services.config.ts:589-607 all …
  • VERIF-AP-T11-006 — Verify: CHANNEL_MODE=simulator in docker-compose.qa-aws.yml · hmp-infrastructure/docker/docker-compose.qa-aws.yml:291 CHANNEL_MODE: "simulator" (with :290 comment to switch to live after ResAvenue creds).
  • VERIF-Plan4-Task20 — Verify: @hmp/api-contracts@1.12.2, simulation-badge.tsx present · simulation-badge.tsx present in hmp-web-dashboard + hmp-admin-portal src/components/ui/. @hmp/api-contracts now 1.14.0 (>=1.12.2 claimed; version adva…
  • channel-providers-crash — admin-portal /channel-providers crash (ApiResponse envelope not unwrapped) · admin-portal CLAUDE.md gotcha 11 (fix/channel-providers-envelope-unwrap 2026-05-22): channel-providers.ts unwraps with .then(r=>r.data.data), fixes e?…

Change log​

DateAuthorChange
2026-06-04Claude/AjayTracker↔board reconciliation (wf_16d0c027). 280/281 items verified vs code+CLAUDE.md: 180 DONE (collapsed into the ✅ CLEARED appendix), 100 still-open (50 OPEN / 22 PARTIAL / 26 stakeholder-BLOCKED / 2 unverifiable), 35 stale labels corrected in place (incl. re-opening FU-3/Plan7-FU3 and P3-8 which were overstated as done). Board: closed 4 false-reds (#6399 AP-T06-005, #6452 AP-T10-007, #6430 FU-44, #6431 FU-47), created 4 blocked launch-gap items (#6454 SES, #6455 NIC-creds, #6456 NIC-webhook, #6457 merchant-account); #6446 B2 kept ops-gated.
2026-05-30 (later)Claude/AjayFU-30 FULLY VERIFIED across all 4 envs + FU-20 bumped to HIGH (prod-blocking). Same-day continuation: applied module.assets_storage to staging (keynova-staging-assets + keynova-staging-property-assets IAM role) and prod (keynova-prod-assets + keynova-prod-property-assets); operator manually ArgoCD-sync'd keynova-staging-hmp-{web-dashboard,property-service} and keynova-prod-hmp-{web-dashboard,property-service} via the ArgoCD UI; smoke test verified end-to-end on each env (upload + list + display + delete). Property #1 admin's original SEV-1 upload bug is fully resolved across all envs. Three blockers discovered + fixed during the rollout: (1) FU-76 (dashboard CSP blocked direct S3 uploads + dev-leak from request.nextUrl.origin in standalone mode) — fix shipped hmp-web-dashboard 62d09c7 → 9210158; (2) FU-77 (private bucket means raw S3 URL 403's on <img src>) — fix shipped hmp-property-service 6a36cbd adds withPresignedGetUrl(row) helper to listImages + registerImage; (3) FU-20 (ArgoCD IDC OIDC login broken — points at HTML signin page, not an OIDC issuer) — workaround used: bootstrap admin password via kubectl get secret argocd-initial-admin-secret; long-term needs SAML 2.0 refactor. Bumped FU-20 from Medium → HIGH given it's actively blocking prod ArgoCD operator access. Also surfaced prod-environment observations (NOT in scope for today, separately tracked): 5 prod Applications OutOfSync from manual-sync drift (auth/frontdesk/payment/api-gateway/reservation — SRE backlog, do NOT bulk-sync), hmp-banquet-service Degraded + Sync failed (needs diagnosis), various 503/400/404/500 noise in dashboard console (notifications IN_APP_ENABLED=OFF → Socket.IO retry storm, /control-center/* stale fetches from deleted endpoints, /api/reports/v2/manager-flash 500 from empty MV per FU-54/FU-55, React error #418 hydration mismatch). FU-30 closure final state: buckets keynova-{test,qa,staging,prod}-assets, roles keynova-{test,qa,staging,prod}-property-assets. Repo HEADs: hmp-infrastructure 22a0710, hmp-property-service 6a36cbd, hmp-web-dashboard 9210158, hmp-docs 4847454. Lesson reinforced from this session: the 2026-05-25 test smoke verified bucket+SA existence but never end-to-end-uploaded through the UI — that gap masked both FU-76 and FU-77. New rule for any future asset-storage rollout: smoke = "upload AND see the tile render AND delete works" via the dashboard UI, not just "infra resources exist".
2026-05-30Claude/AjayFU-30 qa apply SHIPPED + 3 new follow-ups (FU-76 dashboard CSP, FU-77 private-bucket presigned GET, FU-78 future CloudFront). Applied module.assets_storage to qa (keynova-qa-assets bucket, keynova-qa-property-assets IAM role, all 8 resources cleanly via scripts/fu-30-apply-property-assets.sh --yes qa). qa smoke surfaced FU-76: dashboard CSP connect-src emitted https://0.0.0.0:3020 wss://0.0.0.0:3020 instead of the public ALB hostname, blocking the direct browser PUT to the S3 presigned URL. Root cause: Next.js standalone mode with HOSTNAME=0.0.0.0/PORT=3020 makes request.nextUrl.origin reflect the pod's bind address rather than x-forwarded-host. Fix shipped in 2 commits (hmp-web-dashboard 62d09c7 + 9210158): (a) added https://*.s3.ap-south-1.amazonaws.com to connect-src; (b) reconstructed origin from header chain x-forwarded-host → host → nextUrl.host with reject list for 0.0.0.0/localhost/127.0.0.1. Build #31484 promoted to all 4 envs; auto-synced on test+qa, manual sync pending for staging+prod. Test smoke verified upload works. Then test smoke surfaced FU-77: uploaded image tiles 403'd because the FU-30 bucket has all 4 S3 public-access blocks ON (intentional — guest ID photos / signatures will share the bucket), and AssetService.listImages was returning the raw https://...amazonaws.com/<key> URL which the browser couldn't GET. Fix shipped (hmp-property-service 6a36cbd): new withPresignedGetUrl(row) helper returns a 1-hour presigned GET URL keyed on objectKey from every read path (listImages + registerImage). 3 new specs + 27 existing all pass. Build #31487 in flight. FU-78 filed for the long-term CloudFront+OAC solution. Numbering note: my "FU-75" from 2026-05-26 was renamed to FU-76 because an unrelated 2026-05-26 Active Connections shipment had also claimed FU-75 in a concurrent tracker update. Lesson from this work: the 2026-05-25 test smoke verified bucket + SA existence but never end-to-end-uploaded through the UI; doing the full UI-through-display flow as the smoke step would have caught both FU-76 and FU-77 a week earlier. New rule for FU-30 staging+prod and any future asset-storage rollout: the smoke step is "upload a PNG via the dashboard AND see the tile render", not just "bucket exists + SA annotated".
2026-05-26Claude/AjayFU-30 PROPERTY-SERVICE PARTIAL-CLOSED + 7 new follow-ups added (FU-68..FU-74). Multi-session Property #1 admin SEV-1 work and discoveries: (a) FU-30 property-service half SHIPPED — Property #1 admin reported POST /api/assets/upload-url returning 500 in test on 2026-05-25 because the property-service pod had no AWS credentials at all (no IRSA annotation, no static keys). Closed end-to-end: new terraform/modules/assets-storage/ module (hmp-infrastructure 908083e) provisioning the per-env S3 bucket keynova-<env>-assets + IRSA role keynova-<env>-property-assets; per-env Kustomize overlay SA annotation + ConfigMap AWS_S3_BUCKET patch; one-shot apply script scripts/fu-30-apply-property-assets.sh (idempotent, per-env SSO + mgmt-account check, plan-aborts-on-destroy, 3 polish iterations: ANSI color quoting, Windows CRLF strip on profile listing, mgmt-account-not-env-account fix); operator runbook monitoring/runbooks/fu-30-property-assets-irsa-apply.md. Applied + verified end-to-end on test 2026-05-25 (S3 bucket created, SA annotated, pod rolled, dashboard upload promoted). qa/staging/prod apply pending operator action — same one-command script. (b) Dashboard contract drift fix — hmp-web-dashboard fd36dd3 aligned ImageUploadWidget + propertyAssetsApi with the backend RegisterImageDto (4 field-name mismatches: key→objectKey, altText→caption, displayOrder→sortOrder, missing url; widget now takes a single object payload). Also removed the duplicate base64 logo path from Property Settings Photo Gallery — Logo & Colors at /settings/branding is now the canonical surface. (c) Rate-plans form bugs SHIPPED — Property #1 admin then hit Invalid ISO date toast (a9cee61 synthesises T00:00:00.000Z / T23:59:59.999Z) and then expected string, received null (4a32de2 uniform || null → || undefined across 5 optional fields: maxStay, maxAdvanceBooking, derivedFromId, derivationMethod, derivationValue). 7 new FUs added for the patterns surfaced during this work: FU-68 (frontdesk S3 IRSA, ~1h copy-paste of property pattern), FU-69 (guest-service S3 IRSA, same shape), FU-70 (api-contracts schema cleanup: .datetime() → .date() + nullable for clear-able fields), FU-71 (dashboard || null codebase-wide sweep — same shape as FU-67's widespread anti-pattern), FU-72 (test coverage gap on rate-plans page — both bugs shipped because nothing exercised the path), FU-73 (dashboard pipeline auto-trigger consistently fails on master pushes — needs az pipelines run every time across all 3 sessions in 24h), FU-74 (Property Settings 12-tab UX consolidation — Setup Wizard cross-link, Policies/Operational-Policies merge, tab count reduction). FU-30 row mutated (line 301) to reflect property-service partial closure. Lesson logged for future sessions: this is the third time in two days that I forgot to update the master pending tracker concurrently with code changes — the Property #1 Readiness charter explicitly requires "platform-registry updated in the same PR". Adding this row to the change log AND the 7 FU entries is the retrospective fix. Future commits in this work surface should touch the tracker in the same PR.
2026-05-24Claude/AjayFU-52 PROD-VALIDATED end-to-end + FU-65 CLOSED — overnight prod RDS auto-rotation fired clean. Saturday-night-scheduled cron(0 21 ? * SAT *) fired prod RDS rotation at 2026-05-24T03:41:21+05:30 (Sunday early morning). End-to-end chain held: (1) AWS Secrets Manager LastRotated updated; (2) ESO 5-min poll picked up new password within ~34s; ESO syncedResourceVersion bumped 1-…→2-fdd81e25…, SecretSynced: True; (3) Reloader v1.4.5 logged 17 distinct Changes detected in 'rds-master-credentials' of type 'SECRET' in namespace 'hmp-apps'; updated '<svc>' of type 'Deployment' lines at 03:41:55-57 IST (covering analytics, api-gateway, asset, auth, banquet, channel, crm, frontdesk, guest, housekeeping, notification, payment, pos, property, reservation, revenue-strategy, spa — exactly the 17 backends that mount DATABASE_URL from rds-master-credentials); (4) All 17 backend pods rolling-restarted by ~04:40 IST — verified at 12:40 IST showing all 17 at AGE 8h, with frontends (admin-portal/booking-engine/web-dashboard) correctly skipped; (5) https://api.keynova.ai/api/health returned 200 throughout. The 6 previously-missing-annotation backends (analytics/banquet/channel/crm/revenue-strategy/spa) ALL appeared in the 17-restart list — confirming 2026-05-23's annotation-gap fix worked. Cadence reset via aws secretsmanager rotate-secret --rotation-rules AutomaticallyAfterDays=7 --no-rotate-immediately; verified Duration=None, ScheduleExpression=None, LastRotated unchanged, NextRotation=2026-05-31T05:29:59+05:30. Status flips: §6 FU-52 ✅ PROD-VALIDATED; FU-65 ✅ CLOSED; §10 row 10 (Reloader validation) ✅ DONE; §10 footer "next-up" trimmed from 5 → 3 items (Reloader validation + AP-T08-005 both closed). Diagnostic gotcha worth knowing for future cross-cluster work: aws sso login --profile X does NOT change kubectl context — always kubectl config current-context first (burned 30min before realising context was still pointed at keynova-test from yesterday's walkthrough). The FU-46-class SEV-1 (stale token / stale DB creds → 401s everywhere on rotation) is now provably closed at the prod tier. Full evidence: [[project_2026-05-24-fu-52-prod-validated]] in user memory.
2026-05-24Claude/Ajay§13 added — Property #1 Dashboard Readiness (PA/PB/PC/PD). Post-onboarding audit of hmp-web-dashboard IA + self-service surfaces produced a 25-item report (docs/superpowers/specs/2026-05-23-property-admin-dashboard-readiness-report.md). Shipped across 4 batches: P0 (7 items, branch prop1/PA-readiness-p0-batch, master f25fc32), P1 (9 items via 6 commits, prop1/PB-readiness-p1-batch, 00c91a9), P2 (9 items via 4 commits, prop1/PC-readiness-p2-batch, e9f07ea), P3 (5 of 8 follow-ups via 2 commits, prop1/PD-readiness-p3-batch, 76d527b). 28 of 33 worked items closed; 5 deferred (3 backend-blocked, 1 obsolete, 1 codebase-wide refactor). Single-repo batch — no other repos touched. Highlights: tenant Email + SMS + WhatsApp credential CRUD on dashboard (PA-5 + PB-4 — closes a P0 admin-portal handoff bottleneck); channel nomenclature standardized to "Channel" / "Channel Manager" (PC-2 — 35 strings, 17 files); booking-engine theme picker + template-variables editor + booking-host runtime config ported from admin-portal (PB-2/PB-3/PD-2); structured OTA-onboarding request form replaces mailto: empty-state (PB-6). Pipelines #30836 (20260523.5 on P2 HEAD) + #30840 (20260524.1 on P3 HEAD, the canonical final gate).
2026-05-24Claude/AjayNext-20 verify-first consolidation — 9 silent closures + 11 plan items. Parallel-agent verify-first sweep on the next-20 prioritized list found 9 of 20 items already shipped but never flipped: AP-T07-006 (099b3ea), AP-T10-007 (c9b045f + 3fb2c15), AP-T10-008 (6f91f91), AP-T11-004 (1155b7f → 2d887b4), AP-T13-006 (under AP-T13-003 P0 lineage), AP-T06-001 (56223a8 + b55fb8d), AP-T01-006 (placeholder/redundant), AP-T03-003 (PB-2 closed it), FU-56 (ReservationConsumer always wrote it). Spec + plan: docs/superpowers/specs/2026-05-24-next-20-verify-first-execution-design.md + docs/superpowers/plans/2026-05-24-next-20-verify-first-execution-plan.md. Remaining 11 items split into Tier-B finishes (FU-63, AP-T13-005, FU-60, AP-T11-005) + Tier-C builds (AP-T13-007 ex-T13-003, AP-T07-005 KitchenStation, FU-57 9-service sweep, FU-58, AP-T16-004, FU-61, FU-64). New FU-66 added — comprehensive branching+pipeline policy queued for post-launch. Cumulative verify-first savings now ~24+ duplicate picks avoided.
2026-05-23Claude/AjayFU-55-frontdesk CLOSED — dual-emit landed. Read-only investigation confirmed analytics FolioConsumer subscribes to canonical folio.posted/folio.closed but frontdesk-service publishes deprecated folio.charge.posted/folio.settled → folio events reach analytics nowhere → empty revenue/segment/country/room-type/rate-plan MVs. EventTopics enum already defines both names with @deprecated markers, mirroring the POS_KOT_CREATED/POS_KOT_GENERATED precedent. Fix shipped: audit/FU-55-frontdesk-folio-event-dual-emit-and-fields (208e25d → merged to master ad2d036, pipeline #30824). FoliosService.postCharge() now dual-emits FOLIO_CHARGE_POSTED + FOLIO_POSTED (with analytics-segmentation fields: segment/countryCode/roomTypeCode/ratePlanCode/roomNights/taxAmount, all fail-soft). FoliosService.settleFolio() dual-emits FOLIO_SETTLED + FOLIO_CLOSED (field-name remap: totalAmount = totalCharges, closedAt = settledAt, + segment/countryCode). 8/8 specs pass + tsc clean. Empty MVs will populate from the next folio events posted post-deploy. Next-Track-4 cleanup retires the deprecated topics after analytics CIs run green for one release cycle. The mv_revenue_by_source anomaly (140 rows from unknown third source) is logged for a separate read-only follow-up — does NOT gate this fix.
2026-05-23Claude/AjayTest-rot cleanup pass — 187 specs + 3 tsc + 1 latent type-hygiene gap CLOSED across 10 local branches in 5 services; 0 product bugs found. Follow-up to the master-tracker consolidation (entry below). All branches off master, local-only — not pushed. Wave 1 (5 branches, 64 fixed): analytics ad92cd4 (2 — mock Prisma mvGuestSpend.deleteMany delegate); guest 3913bcc (2 — wrong Aadhaar Verhoeff test inputs; product code correct; replaced with UIDAI test number 234123412346); payment 5b014c1 (3 tsc + 1 latent — added payment.captured to PaymentEventName union + providerPublicKey/amount to InitiateResult + tightened outbox WriteEventInput.eventName); reservation 2dfd961 (4 — mockGstService.getRoomGstRate leak + missing cancellationPolicy mock + stale state-machine assertion); frontdesk 3f61ca5+5831cf4+5ae7fdc (49 — CHECKIN-PERF-01 setImmediate + FINDING-003 3-arg + W1.B.3/4 foreign-guest gate + expressCheckIn rewrite + folio refund contract P0-1/P0-3 + NightAuditService 6→10-dep DI drift across 3 night-audit suites). Wave 2 (5 branches, 122 fixed): guest 19c55c3 (12 — GdprService FieldEncryptionService+KafkaService + ReadModelSyncService 4-topic subscribe); frontdesk-controller 34fcc82 (12 — NightAuditController 2→5-dep DI drift); frontdesk-residual 9c874fd+2ceac08+2136871+9ab9e92+5d2ace8 (98 across 10 specs — AuthorizationService GuestServiceClient GSA-060 + tx.folioTransaction.create + bedNumber/qrToken denorm; CheckInService GuestServiceClient + setImmediate; CheckOutService EInvoiceService+EInvoiceJobRepository auto-fire IRN; main.spec EINVOICE_MOCK boot-validator contract; calendar roomTypes-unfiltered + unassignedReservations source filter; keycard 3-attempt retry mockResolvedValueOnce; read-model-sync 6→8 topics; guest-checkin lastName lookup-via-client). Pattern across all 187 failures: 0 product bugs — every one a test bug (DI drift / missing mock delegate / mock-state leak / stale assertion / wrong inputs / contract drift). Product code is healthier than the test corpus. Env-dependent (recommend CI-infra ticket, NOT code fix): reservation-service ~185 failures across 14 e2e/Redis-dependent suites; frontdesk roundtrip.e2e.spec.ts (hard-coded localhost:5438+localhost:6379). AP-T16-004 unblocked once these 10 branches merge. All branches local-only — operator review + push pending.
2026-05-23Claude/AjayMaster-tracker consolidation — gold-standard parallel-agent batch (24 audit items). A priority-ranked next-20 was selected from this tracker; the auto-workable subset was executed by parallel charter-compliant subagents (TDD, verify-first, local audit/* branches, no push). Verified already-shipped (no commit): AP-T01-004, AP-T02-001, AP-T03-001, AP-T11-001, AP-T11-006, Plan 4 Task 20, AP-T01-007 (xPrisma already migrated to PropertyServiceClient cd44244), AP-T04-003 (c7a8acb+3d90b39), AP-T10-001 (aa90ac9), AP-T07-004 producer (099b3ea) + AP-T07-006. Newly fixed — 18 local branches across 9 repos, NOT pushed: AP-T06-002 guest-service 06eb705 (typed GuestVisa model + FRRO/C-Form fields + migration 20260523000000_guest_visa_frro_fields); AP-T08-005 frontdesk eca3feb + notification 5f39f4b (also fixed a broken shipped spec); AP-T08-007 frontdesk e8c4bcd; AP-T12-004 payment e6740bf (new PaymentReconcileService — root-cause layer for never-delivered webhooks); AP-T05-005 8204aa3 + AP-T05-006 8b17553 frontdesk room-move; AP-T07-004 consumer 3054c0f (also shipped a MISSING FolioTransaction GST-columns migration — latent prod crash); AP-T09-003 094e618; AP-T06-005 frontdesk self-checkin de998ff + booking-engine UI 92e2f53; AP-T10-009 analytics 8e82d22; AP-T10-006 analytics c0d9160; AP-T16-002 dashboard 1ad407b (suite now 1592/1592 green — "309 failures" was stale, only 4 real incl. a product bug); AP-T05-004 abb8be5 + AP-T04-004 4b58c2a dashboard (latter fixed a latent tax double-count); FU-55 reservation 866391d + housekeeping e8bf812. Deferred (3, with reasons): AP-T06-002 check-in wiring (needs guest-service merge + @hmp/service-clients GuestServiceClient methods); FU-55-frontdesk folio events (needs an event-contract decision — analytics FolioConsumer subscribes to folio.posted/folio.closed which frontdesk NEVER publishes → root cause of empty revenue/segment MVs); FU-54 (re-evaluate after that decision). Pre-existing test rot found on master (NOT introduced; ~57 failing specs + 3 tsc errors across 5 services; blocks AP-T16-004 — recommend a cleanup pass): reservation-service reservations.service.spec.ts 4, frontdesk checkin.service.spec.ts 9 + folios.service.spec.ts 1 + 3 night-audit suites 39, analytics guest-erased.consumer.spec.ts 2, guest-service guests.service.spec.ts 2, payment-service 3 tsc + payment.captured missing from PaymentEventName. All 18 branches local-only — operator review + push pending.
2026-05-23Claude/Ajay§12 added — prod-onboarding fixes (LOTUS LEAF RESORT). First paying customer onboarded on prod surfaced 18 issues across the launch flows; 16 closed. New §12 summarises A1–F2 with status; full detail in docs/superpowers/plans/2026-05-22-prod-onboarding-fixes.md. Highlights: A5 (booking-engine publishable API key — brainstorm→spec→plan→built across 5 repos, verified live on prod — closes AP-T03-002, §2 + §11 rows flipped ✅) and F1 (gateway rate-limit keying — replaced the spoofable left-most-XFF band-aid with trust proxy + verified-JWT 3-tier keying, built across 3 repos, deploying). Open: C2 (AWS SES re-review) + E1/E2 (ResAvenue go-live, blocked on per-hotel creds). Plan 5 "ResAvenue first hotel onboarding" row updated to point at the new go-live runbook.
2026-05-20Claude/AjayInitial master tracker. Folds in 2026-04-28 audit + 2026-05-08 deferred + 2026-05-09 coverage + 2026-05-17/18 plan checklists + 374 AP-T closure commits across 18 repos.
2026-05-21Claude/AjayDerived-section sync. After the per-row flips landed, the rolled-up summary sections were still stale. Fixed: §2 P0 closure summary (🔴 NOT STARTED ~12 → 0; ✅ DONE ~22 → ~35); §3 AP-T08-006 flipped 🔴 → ✅ (Folio irn/signedQRCode/ackDate/ackNumber columns verified present, migration 20260429120000_add_folio_irn_columns) + P1 summary ~14 → ~15 done; §10 Top-25 table given a Status column — 19 of 25 now ✅, 6 remaining are all operator/stakeholder-gated (FU-51, FU-46, Reloader validation, Op RDS scheduling, AP-T08-005 partial, Q4).
2026-05-21Claude/AjayFU-46 + AP-CC-3 CLOSED. Prod-bootstrap marathon (2026-05-20→21) exposed FU-46 as only half-shipped — ALLOW_HS256 bypassed the boot guard but a request-time guard still demanded RS256, breaking admin login. Fixed end-to-end: (a) scripts/fu-46-provision-env.sh provisions a per-env RS256 keypair to all 4 envs (auth-service private+public, all other backends public), run for test/qa/staging; prod done manually; (b) keypairs folded into Terraform tls_private_key.jwt in all 4 env stacks (5197318) — from-scratch rebuilds now get RS256 natively; (c) ALLOW_HS256 removed from all 16 overlay configmaps (995ec4a); (d) playbook rewritten (2ed6407). staging+qa dashboard login Playwright-verified on RS256. §1, §2 summary, §6, §10 rows all flipped to ✅. Top-25 now 20/25 closed. Related marathon work (not pre-tracked rows, recorded for history): prod got its first SUPER_ADMIN (platform-admin@keynova.ai, password rotated off the seed bcrypt); the frontend NEXTAUTH_SECRET was moved from out-of-band K8s Secrets onto the GitOps ExternalSecret path (plan 2026-05-21-frontend-nextauth-secret-gitops.md, all 4 phases shipped). Still open: prod-demo seed schema drift — seed-prod-demo-{tenant,property,mock-providers}.sql reference columns that no longer exist (subscription_plans.updatedAt, properties.checkInTime, merchant_accounts.provider, provider_configs table); blocks the [DEMO] sanity-check tenant on prod (NOT first-property-blocking).
2026-05-21Claude/AjayProd-demo seed schema drift CLOSED (closes the open item flagged in the row above). All 3 seed-prod-demo-*.sql files diffed against the current Prisma schemas and rewritten to match (hmp-infrastructure 5bf0ed1): (a) seed-prod-demo-tenant.sql — dropped removed subscription_plans.updatedAt, added required maxProperties, features text[]→json; added required tenant_subscriptions.currentPeriodStart/currentPeriodEnd. (b) seed-prod-demo-property.sql — dropped removed properties.checkInTime/checkOutTime; dropped tenantId from room_types/rooms/rate_plans (those tables scope via propertyId now); basePrice→baseRate; quoted rooms.floor (now a String column). (c) seed-prod-demo-mock-providers.sql — provider_configs→provider_credentials; config JSON → placeholder credentialsEncrypted bytea + required scope/billingMode enums; dropped the PUSH row (ProviderChannel has no PUSH value) → 3 rows. Raw-SQL merchant_accounts seeding dropped — the table now stores envelope-encrypted credentials only the payment-service CryptoModule can produce (per hmp-payment-service/CLAUDE.md); the sales engineer creates the sandbox Razorpay account via the dashboard post-seed. Orchestrator seed-prod-demo.sh fixed (3 schema-stale verify spots: rooms.tenantId→propertyId, provider_configs→provider_credentials, notification count 4→3) and the payment seed step removed; Job yaml header + dead DATABASE_URL_HMP_PAYMENTS env removed. Verified against a scratch postgres:16-alpine DB seeded from current-schema DDL — orchestrator reaches "✓ Prod demo tenant ready." and is idempotent on re-run (tenants=1, rooms=30, provider_credentials=3 across EMAIL/SMS/WHATSAPP). Operator step: apply the prod-demo Job on keynova-prod (kubectl -n hmp-apps apply -k kubernetes/jobs/seed-runner/prod-demo/).
2026-05-20 (later)Operator + ClaudeFU-53 + AP-T10-003 operator-executed close-out. FU-53 metrics-server helm upgrade --install ran on test + qa + staging + prod (all 4 envs needed it — staging+prod weren't pre-installed as previously assumed); all 4 envs now show metrics-server-c978df89c-* 1/1 Running + kubectl top nodes returns real values. AP-T10-003 diagnosis run end-to-end via ephemeral postgres:16-alpine pod in kube-system: 9 of 31 MVs populated on QA, FolioConsumer + ReservationConsumer + ChannelConsumer + HousekeepingConsumer + NightAuditConsumer running healthy, 3,310 folio_transactions through today flowing to MVs. EventStore being empty is BY DESIGN (residual schema; actual flow is Redis Streams → consumers → MVs). The original "/api/reports/* returns {data:[]}" symptom was overstated; remaining empty MVs are now tracked as FU-54 (POS/payment demo data) + FU-55 (Plan-2 producer payload fields). No analytics-service engineering fix needed.
2026-05-20Claude/AjayTop-10 closure session (single chat): Verified 5 alleged-shipped items, all confirmed SHIPPED+TESTS_PASS (AP-T05-001/002, AP-T07-001/002, AP-T08-003, AP-T08-004, AP-T09-002). Verified 5 more from the top-10 list also already shipped (AP-T10-002, AP-T10-004, AP-T10-005, AP-T12-003, AP-CC-1). Shipped fresh code in this session: (a) AP-CC-1 P0 drift fix — 4 new @All decorators in proxy.controller.ts for distribution/allocations + dashboard/portfolio; (b) AP-T08-001 stale spec rewrite (auto-invoice-number test re-aligned to current canonical-path contract); (c) L1 graceful shutdown across 20 base Deployments — terminationGracePeriodSeconds: 60 + preStop sleep 10; (d) L2 ALB deregistration_delay=30s on 4 base ingresses + prod HPA minReplicas 2→3 on 6 critical services; (e) FU-53 Terraform module terraform/modules/eks-addons/metrics-server.tf ready for operator apply; (f) AP-T10-003 7-step diagnostic runbook authored at monitoring/runbooks/ap-t10-003-qa-empty-mv-diagnosis.md for operator execution. Verify-first via [[feedback_verify-engineering-debt-claims-in-code]] saved 9 picks of duplicate engineering — biggest single-session save of the saga.
2026-06-09Claude/Ajay§14 added — AP-ARCH architecture remediation track (30 items, ADDITIVE to the 47+41+39). A platform-wide architecture review (performance/security/reliability — platform-registry/architecture-review-2026-06-09.html) produced 29 findings; a remediation plan (superpowers/plans/2026-06-09-architecture-remediation-ap-arch-v2.md) decomposed them into the AP-ARCH-{PERF,SEC,REL}-N series + 2 new items (OPS-1 settlement-folio guard, OPS-2 refund authority tiers) and was reviewed by Technical-Architect + SME + BA agents (all Approve-with-changes; v2 folds in all 14 must-fixes). These are NEW findings, verified NOT duplicates of the April flow-based audit. Highest-impact: a verify-first sweep confirmed the EKS DATABASE_URL has no connection_limit (PERF-1·A), cross-property IDOR on all cash-drawer/petty-cash :id endpoints (SEC-1), committed QA/staging admin creds in git (SEC-2), fire-and-forget reservation lifecycle events lose on crash (REL-1), folio-payment idempotency falls open on Redis outage (REL-2), and the reported folio-lag symptom needs a settlement-time correctness guard (OPS-1), not just latency reduction. Go-live-blocking subset defined in the v2 plan. NOT yet executed — plan + reviews only.

14. AP-ARCH — architecture remediation track​

Added 2026-06-09. ADDITIVE to the 47 P0 + 41 P1 + 39 P2 audit items — these are new architectural findings, not a re-count. Full plan (with go-live gate, owners, rollback, prod success-metrics): superpowers/plans/2026-06-09-architecture-remediation-ap-arch-v2.md. Source review: platform-registry/architecture-review-2026-06-09.html. Reviewed by Technical-Architect + SME + BA (all Approve-with-changes; v2 incorporates the 14 must-fixes).

Go-live-blocking for Property #1 (close before a live guest): SEC-2, SEC-1, REL-1, REL-2, OPS-1, PERF-2, REL-6, REL-3, SEC-7a, SEC-8a, REL-4. Fast-follow (single-property risk-accept): PERF-1·B, PERF-4, REL-5, REL-8, REL-9, REL-10, SEC-4, SEC-7b, SEC-8b, all P2/P3.

IDTitleSevBus. impactEffortRepoStatus
AP-ARCH-SEC-2Remove + rotate committed QA/staging admin credsP0ComplianceSinfra🔴 NOT STARTED
AP-ARCH-PERF-1·Aconnection_limit=5 on EKS DATABASE_URLP0OutageMinfra🔴 NOT STARTED
AP-ARCH-REL-3Kill-switch flag observability (+analytics SKIP_EVENT_SUBSCRIPTIONS)P0OperabilityS–Mreservation+analytics🔴 NOT STARTED
AP-ARCH-SEC-1Cash/petty-cash cross-property IDOR (tenant+role assert)P0MoneyMfrontdesk🔴 NOT STARTED
AP-ARCH-REL-1Lifecycle events → outbox (drop fast-lane)P0Guest-billingMreservation🔴 NOT STARTED
AP-ARCH-REL-2ProcessedFolioEvent idempotency (Redis-outage-safe)P0Guest-billingMreservation🔴 NOT STARTED
AP-ARCH-OPS-1Settlement-time folio correctness guardP0 (new)Guest-billingMfrontdesk🔴 NOT STARTED
AP-ARCH-PERF-2Immediate post-commit relay kick (+kill-flag)P1→blockingGuest-latencyMreservation+frontdesk🔴 NOT STARTED
AP-ARCH-REL-6Night-audit lock: pg_try_advisory_xact_lockP1→blockingGuest-billingMfrontdesk🔴 NOT STARTED
AP-ARCH-REL-4Dead-letter outbox alertingP1→blockingRevenue/opsSreservation+frontdesk+infra🔴 NOT STARTED
AP-ARCH-SEC-7aCERT-In auth brute-force breach producerP1→blockingComplianceSauth🔴 NOT STARTED
AP-ARCH-SEC-8aRuntime money-unit sanity guardP1→blockingGuest-billingSshared-pkgs+frontdesk🔴 NOT STARTED
AP-ARCH-REL-7BaseServiceClient: don't retry writes on reset (after REL-2)P1Guest-billingMshared-pkgs🔴 NOT STARTED
AP-ARCH-PERF-3Availability holds N+1 → batchedP1LatencySreservation🔴 NOT STARTED
AP-ARCH-REL-5Night-audit double-count → UPSERT (propertyId,businessDate)P1InternalSfrontdesk+analytics🔴 NOT STARTED
AP-ARCH-SEC-3WebSocket no-property-claim scope bypassP1Cross-tenant leakMhousekeeping+pos🔴 NOT STARTED
AP-ARCH-SEC-5Prod overlay excludes ALLOW_HS256/MFA-offP1AuthSinfra🔴 NOT STARTED
AP-ARCH-OPS-2Refund/force-checkout amount-tier authorityP1 (new)MoneySfrontdesk🔴 NOT STARTED
AP-ARCH-PERF-1·BWire PgBouncer on EKS (after PERF-1·A, w/ REL-6)P1OutageLinfra🔴 NOT STARTED
AP-ARCH-PERF-4Analytics folio consumer batchingP1InternalManalytics🔴 NOT STARTED
AP-ARCH-REL-11Night-audit chunked + resumable + watchdog (⊃PERF-5)P1Guest-billing/opsMfrontdesk🔴 NOT STARTED
AP-ARCH-REL-8Daily read-model reconciliation sweepP1ConsistencyLfrontdesk+reservation🔴 NOT STARTED
AP-ARCH-REL-9Frontdesk status + balance drift backstopP1Checkout failsMfrontdesk🔴 NOT STARTED
AP-ARCH-REL-10Analytics GuestConsumer stub → MvCountryStatsP1InternalManalytics🔴 NOT STARTED
AP-ARCH-SEC-4Finish assertPropertyTenant rollout (52 sites)P1Cross-tenant leakLmulti🔴 NOT STARTED
AP-ARCH-SEC-6Gateway FTS fail-closedP2Cross-tenant leakSgateway🔴 NOT STARTED
AP-ARCH-SEC-7bCERT-In WAF/payment/DSAR producersP2ComplianceMgateway+payment+guest🔴 NOT STARTED
AP-ARCH-SEC-8bBranded Paise/Rupees + payment.refunded schemaP2Guest-billingSshared-pkgs🔴 NOT STARTED
AP-ARCH-PERF-6Availability via stampede-safe getOrSetP2LatencySreservation🔴 NOT STARTED
AP-ARCH-PERF-7Gateway prefix-trie + single JWT parseP2LatencySgateway🔴 NOT STARTED
AP-ARCH-REL-12Unify @hmp/common circuit-breaker errorFilterP2AvailabilitySshared-pkgs🔴 NOT STARTED
AP-ARCH-REL-13Orphaned-module CI guard + history-drop + chain-audit DBP2OperabilityMfrontdesk+per-svc🔴 NOT STARTED
AP-ARCH-REL-14Case-insensitive feature-flag parsingP2OperabilitySmulti🔴 NOT STARTED