Pending work checklist — 2026-05-20 snapshot
Authored: 2026-05-20.
Supersedes: 2026-05-18-pending-work-checklist.md (preserved for history).
Method: Verified each prior-checklist item in code per feedback_verify-engineering-debt-claims-in-code memory — git log + manifest review across hmp-infrastructure + hmp-docs since 2026-05-18.
How to use:
- Hard blockers at the top — track these to dated deadlines.
- Per-plan sections below — pick up the next-actionable item per plan.
- Cross off (or change status) as items close; commit each closure with the evidence file in
compliance/audit/. - This doc is a snapshot — when the situation changes meaningfully, re-snapshot under a new dated filename rather than mutating this one.
🔴 HARD BLOCKERS (dated deadlines)
| ID | Item | Deadline | Owner | Evidence/Runbook |
|---|---|---|---|---|
| FU-51 | Provision real SendGrid prod live key via AWS SSM keynova/prod/sendgrid + apply hmp-third-party-secrets ExternalSecret. Stub still in kubernetes/overlays/prod/hmp-notification-service/kustomization.yaml. Real email sends will silently 401 until done. | Before first-property onboarding (~2026-06-15) | Operator | hmp-infrastructure/compliance/follow-ups.md#fu-51 |
| FU-46 | RS256 JWT keys for staging+prod; remove ALLOW_HS256: "true" stub from prod overlay configmaps (hmp-api-gateway, hmp-auth-service, hmp-housekeeping-service, hmp-pos-service). scripts/rs256-rotation.sh exists, awaiting operator execution. | Before first-property onboarding | Operator + Eng | hmp-infrastructure/compliance/follow-ups.md#fu-46 + scripts/rs256-rotation.sh |
| Reloader prod validation | Force-rotate test cluster RDS to validate Reloader works end-to-end, then monitor prod natural rotation. Reloader v2.1.5 deployed to all 4 envs 2026-05-17; NOT YET force-tested. | 2026-05-24 (next prod RDS rotation, 4 days away) | Operator | hmp-infrastructure/monitoring/runbooks/fu-52-reloader-rollout.md |
| Plan 6 Q4 | On-call rotation set up with ≥3 named approvers (2-of-N gate requires 3 humans; currently 1) | 2026-06-15 | VP Eng + SRE Lead | hmp-docs/docs/platform-registry/plan-6-decision-log.md#q4 |
| Plan 6 Q5 | Backup retention recommendation sign-off (current 30d prod / 7y CloudTrail satisfies SOC 2 + DPDPA) | 2026-06-15 | Legal/DPO + CISO + SRE Lead | …plan-6-decision-log.md#q5 |
🟡 Plan 4 — Demo tenant on prod (status clarified)
| Task | Status | Action |
|---|---|---|
| Task 24 prod demo-tenant seed | ✅ Job manifest + runbook authored. Auto-seed pipeline (new, see Plan 4-bis below) handles test+qa equivalent. Prod execution STILL gated by 2-of-N approval per monitoring/runbooks/plan-4-seed-prod-demo.md. | Operator: get 2-of-N approval → apply via kubectl |
| Plan 4-bis Asset-mgmt demo data enrichment (3 phases) | ✅ SHIPPED 2026-05-20 — all 4 envs. Includes PPM schedules, AMC-asset links, vendor encryption-safety, Brand/Model/SerialNo/PurchaseDate/PurchaseCost on demo assets. | — |
🟡 Plan 5 — Cutover (in 7-day burn-in)
| Item | Status | Action |
|---|---|---|
api.keynova.ai + live.keynova.ai | ✅ Live on prod EKS ALB (verified 2026-05-17 with ResAvenue handler 400-on-bad-XML) | — |
| 7-day prod burn-in observability | 🟢 In flight — day 3 of 7 (cutover 2026-05-17; closes 2026-05-24) | Daily check via monitoring/runbooks/resavenue-cert-reissuance.md "7-day burn-in" section |
| ResAvenue first hotel onboarding | 🟡 Waiting on first hotel's data | Operator: replace placeholders in seeds/seed-resavenue-prod-connection.sql + apply |
| FU-6 ResAvenue cert v3 closure | 🟡 Closes after burn-in completes | ~2026-05-24 |
🟡 Plan 6 — Prod-launch baseline (no progress since 2026-05-18)
| ID | Item | Status | Action |
|---|---|---|---|
| Baseline re-run | 🔴 Operator must re-run YAML after AWS profiles + flags fixed → expect 17-22 PASS | Operator | |
| B-17 | Renovate.json propagation to 25 repos | 🟡 Template + script authored | Operator runs scripts/propagate-renovate-config.sh |
| B-18 | First quarterly access review | 🟡 Runbook + 2026-Q2 schedule shipped | First review due 2026-06-30 |
| B-19 | DR drill execution (failover ap-south-1 → ap-south-2) | 🔴 Not executed | Operator (SRE) — author DR drill script next |
| B-20 | Backup-restore drill against keynova-prod-vault | 🔴 Not executed | Operator (SRE) |
| B-21 | Pentest report | 🔴 Vendor not engaged | External — 2-4 week procurement lead time |
| B-23 | CIC-1..5 sign-offs (all currently signed=0) | 🔴 Open | Product / Security / SRE / CS / Payments stakeholders |
| B-24 | Spec §14 5 decisions (see Plan 6 decision log) | 🟡 Recommendations drafted | Sign-offs pending (Q4+Q5 are hard blockers above) |
🟡 Plan 7 — Legacy retirement (in progress)
| Phase | Status | Action |
|---|---|---|
| Phase 1 inventory + 2-of-N approval | ✅ Authored | — |
| Phase 2 data-snapshot | Skipped per operator confirmation (seed-only data) | — |
| Phase 3 destroy | 🟢 Operator manually deleting legacy account 730335256638 via AWS Console (in progress 3+ days) | Continue Console deletions; small footprint (1 EC2, 1 RDS, 1 Redis, 1 S3, 1 Secret) |
Phase 4 code cleanup (git rm legacy Terraform stacks + compose files + deploy scripts) | 🟡 PENDING — terraform/{pilot-aws,staging-aws,prod-aws}/ directories STILL EXIST (verified 2026-05-20) | Author single cleanup commit when Phase 3 done |
| Phase 5 cost verification (T+7d post-destroy) | 🟡 Pending | Operator pulls AWS Cost Explorer 7 days after final destroy |
| FU-3 legacy module dedup | ✅ Closes when Phase 4 commits | — |
🔵 Engineering hardening for scale (preparing for 100s of hotels) — UNCHANGED
All levels still 🔴 NOT STARTED. Verified 2026-05-20: no preStop hooks, no terminationGracePeriodSeconds, no deregistration_delay annotations, no HPA minReplicas bumps, no RDS IAM auth migration in app code.
| Level | Item | Why | Effort | Status |
|---|---|---|---|---|
| L1 | Add preStop hook (sleep 10) to 20 base Deployments | Eliminates ALB target-deregistration race → ~95% of rolling-restart 502s gone | 30 min | 🔴 Not started |
| L1 | Bump terminationGracePeriodSeconds to 60s on all Deployments | Cleaner pod shutdown alignment with ALB drain | 15 min (same commit) | 🔴 Not started |
| L2 | Tune ALB deregistration_delay.timeout_seconds=30 (currently default 300) | Faster target removal; tighter rolling-restart window | 30 min (per-Ingress annotation) | 🔴 Not started |
| L2 | Bump prod HPA minReplicas from 2 to 3 on critical services | Rotation window only affects 1/3 not 1/2 | 30 min | 🔴 Not started |
| L3 | Migrate to RDS IAM auth | Eliminates password-rotation impact entirely — Plan 2 already enabled enabled_iam_authentication = true on prod RDS; app needs to use it | ~2 days engineering | 🔴 Not started |
| L4 | AWS RDS Proxy (optional) | Transparent rotation + pool; +~$15/mo per proxy | ~1 day | 🔴 Not started |
| Op | Schedule prod RDS rotation for off-peak (02:00 IST) | Operator-controlled timing | 1h ops setup | 🔴 Not started |
Recommendation: L1 + L2 land BEFORE first-property onboarding. L3 lands before scaling beyond 10 hotels.
🟡 Other open follow-ups (compliance/follow-ups.md)
No new FU entries since 2026-05-17 23:46. Current list unchanged from 2026-05-18:
| ID | Item | Owner |
|---|---|---|
| FU-2 | Tooling installation runbook (tflint / tfsec / checkov local install guide) | Plan 0 carry-over |
| FU-5 | AWS Config delivery channel + S3 Object Lock incompatibility (needs separate bucket) | Plan 6 |
| FU-11 | IDC OIDC ArgoCD app — operator console registration | Operator |
| FU-12 | ArgoCD per-env PAT population in Secrets Manager | Operator |
| FU-13/14 | Windows DNS poisoning + TCP parallelism | Workaround in place |
| FU-18 | Kustomize v5 namespace transformer ID conflict | Architecture follow-up |
| FU-19 | IDC group→permission-set assignments don't include mgmt account | Operator |
| FU-20 | ArgoCD SSO needs SAML 2.0 refactor (FU-11 blocked path) | Operator |
| FU-21 | A.1 stored bot PAT in mgmt account; ESO IRSA is env-local | Operator |
| FU-22 | .gitattributes did not cover .tf files | Workaround applied |
| FU-23 | Mgmt-account Azure DevOps OIDC + deploy role wasn't created in Plan 0 | Resolved via gap-fill stack |
| FU-24 | Hotfix pipeline assumes Jira; we don't have one | Plan 6 hardening |
| FU-25 | Migrate deploy notifications Slack → Teams | Q3 sign-off + operator config |
| FU-26 | ArgoCD ci-bot tokens are non-expiring + no revocation runbook | Plan 6 hardening |
| FU-27 | ArgoCD helm_release timeout false-negative | Workaround documented |
| FU-28 | Self-approval allowed on staging/prod during pilot | Closes when Q4 lands |
| FU-29 | Plan 3 T27/T28 per-service overlay subdirectories | Resolved (Plan 3 SHIPPED) |
| FU-30 | Per-service ServiceAccounts missing IRSA annotations | Plan 6 hardening |
| FU-32 | Plan 3 templates had trigger:/pr: blocks | Resolved |
| FU-43 | Per-env auth-service-runtime-env propagation | Operator runtime secret rotation |
| FU-44 | @hmp/cache + @hmp/messaging TLS support | ✅ Resolved per recent commits |
| FU-45 | Service runtime secret rotation policy + Lambda | Plan 6 hardening |
| FU-46 | RS256 JWT keys for staging+prod (also hard blocker above) | Operator |
| FU-47 | @hmp/observability HealthService paramtypes broken | ✅ Resolved per recent commits |
| FU-48 | Asset-service migration history reset | Operator playbook at monitoring/runbooks/fu-48-asset-migration-reset.md |
| FU-49 | Notification-service device_tokens orphan migration | Same playbook as FU-48 |
| FU-50 | promote-image.sh git push rebase-retry | ✅ Resolved (2026-05-17) |
| FU-52 | RDS auto-rotation breaks pods (Reloader fix) | ✅ Deployed 2026-05-17; validation due 2026-05-24 |
| FU-51 | Real SendGrid prod key (also hard blocker above) | Operator |
| FU-53 | metrics-server install on test + qa | Engineering authoring (~30 min) — mirror FU-52 pattern |
✨ NEW since 2026-05-18
Auto-seed pipeline (ci-cd/azure-pipelines/auto-seed-test-qa.yml) — SHIPPED + LIVE
| Item | Status | Notes |
|---|---|---|
| Auto-seed pipeline for test + qa | ✅ LIVE 2026-05-19 | Trigger: changes to seeds/** or kubernetes/jobs/seed-runner/** on master. Runs in-cluster seed-runner Job via kubectl apply. Test + qa only — hard guards prevent staging/prod execution (separate service connections + context checks). |
batch: true gate added | ✅ 2026-05-20 (e497667) | Prevents concurrent-update races when manual trigger collides with CI-triggered run |
| Server-side apply for ConfigMap (eb02d72) | ✅ Fixes 262KB annotation limit | seed-runner uses kubectl apply --server-side for the seeds ConfigMap |
| Idempotent ConfigMap create-then-apply (3dc465c) | ✅ Race-safe ConfigMap updates | — |
Asset-mgmt demo data enrichment (Plan 4-bis, 3 phases)
Shipped 2026-05-20 across all 4 envs:
- Phase 1: PPM schedules + AMC-asset links + vendor encryption-safety (c1465cb)
- Phase 2: Brand/Model/SerialNo/PurchaseDate/PurchaseCost on demo assets (2b9deb1)
- Phase 3: consolidated merge + documentation (6e8375a)
2 minor self-fixes (no broken state remaining)
- cbce524 (2026-05-19 revert) — duplicate SQL files from a parallel merge; reverted cleanly. Sourced the
verify-engineering-debt-claims-in-codememory update ("git log BEFORE re-authoring" rule). - e497667 (2026-05-20) — auto-seed pipeline batching to prevent concurrent-trigger collisions. Fix not a regression.
🟡 Sales & Marketing sign-offs (per 2026-05-16 playbook) — UNCHANGED
| Item | Owner |
|---|---|
| VP Sales + CFO sign-off on pricing tiers | Stakeholder |
| CMO + CEO sign-off on marketing plan | Stakeholder |
| Board sign-off on KPIs + org/comp plan | Stakeholder |
Reference: hmp-docs/docs/sales-and-marketing/ (25 files, 10 sections).
🔴 Property #1 Readiness Audit (2026-04-28 verdict: NO-GO) — STILL NOT TRIAGED
| Severity | Count | Status |
|---|---|---|
| P0 | 47 | 🔴 Closure rate STILL not tracked in this snapshot |
| P1 | 41 | 🔴 |
| P2 | 39 | 🔴 |
Mandatory reading per project CLAUDE.md:
hmp-docs/docs/platform-registry/property-1-readiness-audit-2026-04-28.mdhmp-docs/docs/platform-registry/property-1-gold-standard-fix-plan-2026-04-28.md
Recommendation: schedule a dedicated session to triage current closure rate + prioritize remaining items relative to first-property timing (now <4 weeks away).
✅ Recently completed (full visibility into past 7 days)
| Item | Date | Notes |
|---|---|---|
| Asset-mgmt demo enrichment (3 phases) | 2026-05-18 to 2026-05-20 | All 4 envs; PPM/AMC/vendor encryption-safety + asset detail fields |
| Auto-seed pipeline for test+qa | 2026-05-19 | New auto-seed-test-qa.yml removes operator friction on demo iteration |
| Auto-seed batching race-fix | 2026-05-20 | batch: true gate prevents concurrent-trigger collisions |
| Plan 3 D.2 (CI/CD pipelines, 24/24) | 2026-05-17 | All envs deploying via Azure Pipelines |
| Plan 5 PROD DNS cutover | 2026-05-17 | api.keynova.ai + live.keynova.ai verified end-to-end |
| FU-52 Reloader deployed all 4 envs | 2026-05-17 18:06-18:13 IST | Closes 2026-05-24 prod RDS rotation risk |
| FU-50 promote-image.sh rebase-retry | 2026-05-17 | Self-tested live during session |
| Plan 6 baseline checklist execution | 2026-05-17 | 1 PASS / 7 FAIL / 13 NOCREDS / 2 AUTOMODE |
| FU-7 Kyverno rewrite | 2026-05-11 | Cluster-wide enforcement restored |
| Plan 4 Tasks 17/19/20/21 | 2026-05-13 | Foundation + 5 cross-service guards |
| Schema/Seed Audit Phases A/B/C/D | 2026-05-15 | 9 migrations + 3 CI gates + 13 demo-data generators |
| Sales & Marketing playbook folder | 2026-05-16 | 25 files / 10 sections / ~5k lines |
Top 5 recommended next actions (priority order — refined for 2026-05-20)
- Force-rotate test cluster to validate Reloader works end-to-end before 2026-05-24 prod rotation (10 min, zero customer risk)
- Run
scripts/rs256-rotation.shto provision RS256 keys + remove ALLOW_HS256 stub (FU-46 — hard blocker for first-property) - Provision real SendGrid prod key via Secrets Manager (FU-51 — hard blocker for first-property)
- Ship Level 1 hardening — preStop hooks + grace period on all 20 Deployments (1h authoring; closes ~95% of rolling-restart 502s)
- Complete Plan 7 Phase 3 + commit Phase 4 cleanup — small footprint remaining; gets dual-spend off our books
Re-snapshot triggers
Re-author this checklist (as a new dated file, don't mutate this one) when any of these happen:
- 2026-05-24 — prod RDS rotation event validates or invalidates FU-52 Reloader fix
- Any hard blocker closes (FU-51 / FU-46 / Q4 / Q5)
- Plan 7 Phase 3 destroy completes — reflect Plan 7 progress + close FU-3
- Property #1 Readiness Audit re-triaged — new P0/P1/P2 numbers
- Plan 4 Task 24 prod demo seed executes — closes Plan 4 prod side
- Plan 5 7-day burn-in completes 2026-05-24 — closes Plan 5
- Any L1/L2/L3/L4 hardening lands — flip status + reassess priorities
- First-property onboarding kicks off (~2026-06-15) — re-baseline against actual first-customer needs