Phase 1 Night Audit Manual Test Plan (QA EC2)

Workstream E3 — Operational test plan for the QA team to execute night audit against the 5-property KeyNova test seed on the QA EC2 host 43.204.247.178. Generated 2026-04-06. Companion document: phase1-night-audit-scenarios.md.

Read scenarios doc first. This plan assumes you understand the 14 test scenarios and the 3 critical bugs that block several of them.

1. Pre-Flight Checklist

Run these commands before starting any audit. If any check fails, stop and escalate.

1.1 Connect to QA host

ssh -i ~/.ssh/keynova-qa.pem ubuntu@43.204.247.178

1.2 Verify all 13 backend services are running

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep hmp-

Expected: 13 rows, all Up. Critical for night audit:

hmp-frontdesk-service
hmp-property-service
hmp-reservation-service
hmp-pos-service
hmp-housekeeping-service
hmp-analytics-service
hmp-api-gateway

1.3 Verify the 5-property seed is loaded

docker exec -i hmp-postgres psql -U hmp -d hmp_property <<'EOF'
SELECT id, name, code, timezone, settings->>'businessDate' AS business_date
  FROM properties
 WHERE id::text LIKE '15000000-%'
 ORDER BY name;
EOF

Expected: 5 rows (Mumbai, Bangalore, Delhi, Jaipur, Goa). All timezone = 'Asia/Kolkata'. The business_date may be NULL on the very first run — that is expected and the first audit will populate it.

If the result is empty, run the seed first:
bash hmp-infrastructure/seeds/seed-master.sh

1.4 Verify there is at least 1 in-house reservation per property

docker exec -i hmp-postgres psql -U hmp -d hmp_reservation <<'EOF'
SELECT
  "propertyId",
  count(*) FILTER (WHERE status = 'CHECKED_IN') AS checked_in,
  count(*) FILTER (WHERE status = 'CONFIRMED' AND "arrivalDate" < CURRENT_DATE) AS no_shows,
  count(*) AS total
  FROM reservations
 WHERE "propertyId"::text LIKE '15000000-%'
 GROUP BY "propertyId"
 ORDER BY "propertyId";
EOF

Expected: each of the 5 properties has at least 1 CHECKED_IN row. If a property has 0, the audit will still run successfully but roomsPosted = 0.

1.5 Confirm no audit is currently running (Redis lock check)

docker exec -it hmp-redis redis-cli KEYS 'audit:lock:*'

Expected: empty. If any locks are present, either an audit is genuinely running (check nightAuditLog) or a stale lock from a previous run needs to be cleared:

docker exec -it hmp-redis redis-cli DEL 'audit:lock:15000000-0000-4000-a000-000000000001'

1.6 Capture a baseline snapshot

Save these counts to pre-audit-baseline.txt for diff comparison after the audit runs:

docker exec -i hmp-postgres psql -U hmp -d hmp_frontdesk <<'EOF' > pre-audit-baseline.txt
\echo === folio_transactions count by code, today ===
SELECT "transactionCode", count(*) FROM folio_transactions
 WHERE "postingDate" = CURRENT_DATE
 GROUP BY "transactionCode" ORDER BY "transactionCode";
\echo
\echo === night_audit_logs today ===
SELECT "propertyId", status, "roomsPosted", "noShowsProcessed", "totalRevenue"
  FROM night_audit_logs
 WHERE "auditDate" = CURRENT_DATE;
EOF

2. Authentication

Get a JWT for the test tenant admin user.

curl -s -X POST http://localhost:3010/api/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"manager@p-mum.keynova.test","password":"Test@KeyNova123!"}' \
  | jq -r '.accessToken' > /tmp/jwt.txt

cat /tmp/jwt.txt | head -c 80 ; echo

Expected: a long JWT prefixed with eyJ.... Stash for later requests:

export JWT=$(cat /tmp/jwt.txt)
export TENANT_ID=00000000-0000-4000-a000-000000000099
export MUMBAI_ID=15000000-0000-4000-a000-000000000001
export BLR_ID=15000000-0000-4000-a000-000000000002
export DEL_ID=15000000-0000-4000-a000-000000000003
export JAI_ID=15000000-0000-4000-a000-000000000004
export GOA_ID=15000000-0000-4000-a000-000000000005

3. Test Execution

Test 3.1 — Pre-audit validation (Scenario 13)

curl -s http://localhost:3010/api/night-audit/$MUMBAI_ID/validate \
  -H "Authorization: Bearer $JWT" \
  -H "x-tenant-id: $TENANT_ID" \
  -H "x-property-id: $MUMBAI_ID" | jq

Expected: {"warnings":[...]}. Note any warnings.

curl -s http://localhost:3010/api/night-audit/$MUMBAI_ID/pre-audit-report \
  -H "Authorization: Bearer $JWT" \
  -H "x-tenant-id: $TENANT_ID" \
  -H "x-property-id: $MUMBAI_ID" | jq

Expected: {"warnings":[...], "counts":{...}, "canProceed":true|false}.

Pass: the response is valid JSON with the documented shape. Fail: HTTP 401 (re-login), 500 (check service logs), or empty response.

Test 3.2 — Single-property audit (Scenario 1)

curl -s -X POST http://localhost:3010/api/night-audit/$MUMBAI_ID/start \
  -H "Authorization: Bearer $JWT" \
  -H "x-tenant-id: $TENANT_ID" \
  -H "x-property-id: $MUMBAI_ID" \
  -H 'Content-Type: application/json' \
  -d '{}' | tee /tmp/mum-audit.json | jq

Expected response (HTTP 201):

{
  "auditLogId": "some-uuid",
  "roomsPosted": <N>,
  "noShowsProcessed": <M>,
  "totalRevenue": <number>,
  "occupancy": <number>,
  "adr": <number>,
  "revpar": <number>,
  "errors": []
}

Capture the audit log ID:

export MUM_AUDIT_ID=$(jq -r '.auditLogId' /tmp/mum-audit.json)
echo "Mumbai audit log: $MUM_AUDIT_ID"

Test 3.3 — Verify Mumbai audit committed correctly

docker exec -i hmp-postgres psql -U hmp -d hmp_frontdesk <<EOF
\echo === audit log row ===
SELECT id, status, "auditDate", "roomsPosted", "noShowsProcessed",
       "totalRevenue", occupancy, adr, revpar
  FROM night_audit_logs
 WHERE id = '$MUM_AUDIT_ID';

\echo === room charges posted today ===
SELECT count(*), sum(amount + COALESCE("taxAmount", 0)) AS total_inc_tax
  FROM folio_transactions
 WHERE "postingDate" = CURRENT_DATE
   AND "transactionCode" = 'ROOM'
   AND "isVoid" = false;

\echo === folios for Mumbai-only ===
SELECT f.id, f."totalCharges", f."totalTax", f.balance
  FROM folios f
  JOIN reservations r ON r.id = f."reservationId"
 WHERE r."propertyId"::text = '$MUMBAI_ID';
EOF

Pass: audit log row has status = 'COMPLETED'. Folio transaction count matches roomsPosted from the response. Folio totals updated. Fail: any of these mismatches; status = RUNNING (means side-effect bug hit); status = FAILED (check errors column).

Test 3.4 — Verify other properties were untouched (Isolation Test)

docker exec -i hmp-postgres psql -U hmp -d hmp_frontdesk <<EOF
SELECT "propertyId", count(*) FILTER (WHERE "auditDate" = CURRENT_DATE) AS today_audits
  FROM night_audit_logs
 WHERE "propertyId"::text LIKE '15000000-%'
 GROUP BY "propertyId"
 ORDER BY "propertyId";
EOF

Expected: only Mumbai (...0001) shows today_audits = 1. The other 4 must show 0.

docker exec -i hmp-postgres psql -U hmp -d hmp_property <<EOF
SELECT id, settings->>'businessDate' AS business_date,
       settings->>'checkInTime'  AS check_in_time,
       settings->>'checkOutTime' AS check_out_time
  FROM properties
 WHERE id::text LIKE '15000000-%'
 ORDER BY id;
EOF

Critical observation point — Critical Bug 3: check whether check_in_time and check_out_time are still '14:00' and '12:00' for Mumbai. If they are now NULL while the other 4 properties still have them, ** Critical Bug 3 has been confirmed** and you must escalate immediately. Mumbai's businessDate should also have advanced by 1 day.

Test 3.5 — Idempotency (Scenario 4)

Re-run the same audit:

curl -i -X POST http://localhost:3010/api/night-audit/$MUMBAI_ID/start \
  -H "Authorization: Bearer $JWT" \
  -H "x-tenant-id: $TENANT_ID" \
  -H "x-property-id: $MUMBAI_ID" \
  -H 'Content-Type: application/json' \
  -d '{}'

Expected: HTTP 400, body {"message":"Night audit already completed for today",...}.

Pass: 400 returned. Verify no new folio_transactions:

docker exec -i hmp-postgres psql -U hmp -d hmp_frontdesk -c \
  "SELECT count(*) FROM folio_transactions WHERE \"postingDate\" = CURRENT_DATE AND \"transactionCode\" = 'ROOM';"

Should match the count from step 3.3.

Test 3.6 — Chain audit across all 5 properties (Scenario 2)

Warning: this requires Mumbai's audit to not already exist for today. If you ran Test 3.2, you must first roll back Mumbai's audit:
curl -s -X POST http://localhost:3010/api/night-audit/$MUM_AUDIT_ID/rollback \
  -H "Authorization: Bearer $JWT" -H "x-tenant-id: $TENANT_ID"
Mumbai will be auto-skipped if not rolled back (chain orchestrator handles "already completed" gracefully).

curl -s -X POST http://localhost:3010/api/night-audit/chain \
  -H "Authorization: Bearer $JWT" \
  -H "x-tenant-id: $TENANT_ID" \
  -H 'Content-Type: application/json' \
  -d '{}' | tee /tmp/chain-audit.json | jq

Expected response (HTTP 201): ChainAuditResult with totalProperties: 5, completed: 5 (or 4 + 1 skipped if Mumbai was already done).

Verify all 5 audit log rows exist:

docker exec -i hmp-postgres psql -U hmp -d hmp_frontdesk <<'EOF'
SELECT "propertyId", status, "roomsPosted", "noShowsProcessed", "totalRevenue"
  FROM night_audit_logs
 WHERE "auditDate" = CURRENT_DATE
   AND "propertyId"::text LIKE '15000000-%'
 ORDER BY "propertyId";
EOF

Expected: 5 rows, all COMPLETED.

Test 3.7 — Cross-property folio leakage check (Concern #6/#7)

This is the most important multi-property assertion. After the chain audit, verify each property's folio transactions are scoped to that property only.

docker exec -i hmp-postgres psql -U hmp -d hmp_frontdesk <<'EOF'
SELECT
  r."propertyId",
  count(*) AS room_charges,
  sum(ft.amount) AS total_revenue
  FROM folio_transactions ft
  JOIN folios f ON f.id = ft."folioId"
  JOIN reservations r ON r.id = f."reservationId"
 WHERE ft."postingDate" = CURRENT_DATE
   AND ft."transactionCode" = 'ROOM'
   AND ft."isVoid" = false
 GROUP BY r."propertyId"
 ORDER BY r."propertyId";
EOF

Compare each row's room_charges against the corresponding night_audit_logs row's roomsPosted. They must match exactly per property. Any mismatch indicates a property-isolation bug.

Test 3.8 — POS reconciliation safeguard (Scenario 6)

This test is currently expected to fail due to Critical Bug 1. Run it to confirm the bug, then escalate.

docker logs hmp-frontdesk-service --since 5m 2>&1 | grep -i 'pOSOrder\|aggregate\|safeguard\|night.*audit' | tail -50

Expected (with bug present): a stack trace mentioning Cannot read properties of undefined (reading 'aggregate') near the pOSOrder reference. Capture the exact error and add it to the bug ticket.

Test 3.9 — Resume failed audit (Scenario 9)

Inject a fault on Jaipur. The cleanest reproducer is to drop the postgres connection mid-transaction. Use pg_terminate_backend:

# Terminal A: start the audit
curl -s -X POST http://localhost:3010/api/night-audit/$JAI_ID/start \
  -H "Authorization: Bearer $JWT" \
  -H "x-tenant-id: $TENANT_ID" \
  -H 'Content-Type: application/json' \
  -d '{}' &
AUDIT_PID=$!

# Terminal B: kill all connections to hmp_frontdesk for the next 3 seconds
docker exec -i hmp-postgres psql -U hmp -d postgres <<'EOF'
SELECT pg_terminate_backend(pid)
  FROM pg_stat_activity
 WHERE datname = 'hmp_frontdesk'
   AND application_name LIKE '%hmp%';
EOF

wait $AUDIT_PID

Expected: the curl returns HTTP 500 with a postgres connection error. The audit log row should be in FAILED state:

docker exec -i hmp-postgres psql -U hmp -d hmp_frontdesk <<EOF
SELECT id, status, errors
  FROM night_audit_logs
 WHERE "propertyId"::text = '$JAI_ID'
   AND "auditDate" = CURRENT_DATE;
EOF

Now resume:

export JAI_AUDIT_ID=<the failed audit id from above>
curl -s -X POST http://localhost:3010/api/night-audit/$JAI_AUDIT_ID/resume \
  -H "Authorization: Bearer $JWT" \
  -H "x-tenant-id: $TENANT_ID" -d '{}' | jq

Expected: HTTP 201. Verify no double-posting:

docker exec -i hmp-postgres psql -U hmp -d hmp_frontdesk <<EOF
SELECT count(*)
  FROM folio_transactions ft
  JOIN folios f ON f.id = ft."folioId"
  JOIN reservations r ON r.id = f."reservationId"
 WHERE r."propertyId"::text = '$JAI_ID'
   AND ft."postingDate" = CURRENT_DATE
   AND ft."transactionCode" = 'ROOM'
   AND ft."isVoid" = false;
EOF

The count must equal Jaipur's roomsPosted from the resumed audit's response — not double.

4. Rollback Procedure (If Audit Goes Wrong)

If you observe any unexpected charges or want to reset the audit to start over:

4.1 Rollback a single audit

curl -s -X POST http://localhost:3010/api/night-audit/$MUM_AUDIT_ID/rollback \
  -H "Authorization: Bearer $JWT" \
  -H "x-tenant-id: $TENANT_ID" -d '{}'

This sets all transactionCode = 'ROOM' rows for the audit date to isVoid = true and reverts no-show reservations to CONFIRMED.

Caveat (gap): rollback does not decrement folio.totalCharges, folio.totalTax, or folio.balance. After a rollback, manually recompute these by re-summing non-void transactions:

UPDATE folios f SET
  "totalCharges" = COALESCE((SELECT sum(amount) FROM folio_transactions WHERE "folioId" = f.id AND "transactionType" = 'CHARGE' AND "isVoid" = false), 0),
  "totalTax" = COALESCE((SELECT sum("taxAmount") FROM folio_transactions WHERE "folioId" = f.id AND "isVoid" = false), 0),
  "balance" = COALESCE((SELECT sum(CASE WHEN "transactionType" = 'CHARGE' THEN amount + COALESCE("taxAmount",0) WHEN "transactionType" = 'PAYMENT' THEN -amount ELSE 0 END) FROM folio_transactions WHERE "folioId" = f.id AND "isVoid" = false), 0)
 WHERE f.id IN (SELECT DISTINCT "folioId" FROM folio_transactions WHERE "postingDate" = CURRENT_DATE);

4.2 Hard reset (last resort — destroys all today's audit data)

docker exec -i hmp-postgres psql -U hmp -d hmp_frontdesk <<'EOF'
BEGIN;
UPDATE folio_transactions SET "isVoid" = true, "voidReason" = 'Manual hard reset of night audit'
 WHERE "postingDate" = CURRENT_DATE AND "transactionCode" IN ('ROOM', 'NO_SHOW');
DELETE FROM night_audit_logs WHERE "auditDate" = CURRENT_DATE AND "propertyId"::text LIKE '15000000-%';
COMMIT;
EOF

docker exec -i hmp-postgres psql -U hmp -d hmp_reservation <<'EOF'
UPDATE reservations SET status = 'CONFIRMED', "noShowAt" = NULL
 WHERE status = 'NO_SHOW' AND "noShowAt" >= CURRENT_DATE
   AND "propertyId"::text LIKE '15000000-%';
EOF

# Clear Redis locks
docker exec -it hmp-redis redis-cli --scan --pattern 'audit:lock:*' | xargs -I{} docker exec -it hmp-redis redis-cli DEL {}

Do not run this on a production database. Hard reset is for QA only.

4.3 Restore property settings (recover from Critical Bug 3)

If Critical Bug 3 has erased checkInTime/checkOutTime from a property, restore from the seed:

docker exec -i hmp-postgres psql -U hmp -d hmp_property <<'EOF'
UPDATE properties SET settings = jsonb_build_object(
  'checkInTime',     '14:00',
  'checkOutTime',    '12:00',
  'defaultCurrency', 'INR',
  'timezone',        'Asia/Kolkata',
  'businessDate',    settings->>'businessDate'
)::json
 WHERE id::text LIKE '15000000-%';
EOF

5. Where to Find Logs

What	Where
Frontdesk service stdout	`docker logs hmp-frontdesk-service --since 1h`
Night audit specifically	`docker logs hmp-frontdesk-service 2>&1 \| grep -i 'night.*audit'`
Postgres slow queries / errors	`docker logs hmp-postgres --since 1h \| grep -iE 'ERROR\|fatal'`
Redis lock activity	`docker exec -it hmp-redis redis-cli MONITOR` (Ctrl-C to stop)
Kafka events for `night.audit.completed`	`docker exec -it hmp-kafka kafka-console-consumer --topic night.audit.completed --from-beginning --max-messages 10 --bootstrap-server localhost:9092`
Loki / centralized logs (if enabled)	`http://43.204.247.178:3100` or via Grafana
Audit log table	`SELECT * FROM night_audit_logs ORDER BY "startedAt" DESC LIMIT 10;`

Common log greps

# All night audit activity in the last hour
docker logs hmp-frontdesk-service --since 1h 2>&1 | grep -iE 'NightAudit|night.*audit|audit.lock'

# Errors only
docker logs hmp-frontdesk-service --since 1h 2>&1 | grep -iE 'ERROR|FATAL|Cannot read|undefined'

# Kafka events fired
docker logs hmp-frontdesk-service --since 1h 2>&1 | grep -iE 'kafkaService.publish|night.audit.completed|reservation.no_show'

6. Sign-off Checklist

After running all tests above, fill in this report and hand back to the production launch owner:

Test	Status	Notes
1.x Pre-flight checks	[ ] PASS / [ ] FAIL
3.1 Pre-audit validation	[ ] PASS / [ ] FAIL
3.2 Single-property audit	[ ] PASS / [ ] FAIL
3.3 Audit log committed	[ ] PASS / [ ] FAIL
3.4 Other properties untouched	[ ] PASS / [ ] FAIL	Confirm Critical Bug 3
3.5 Idempotency	[ ] PASS / [ ] FAIL
3.6 Chain audit	[ ] PASS / [ ] FAIL
3.7 No cross-property folio leakage	[ ] PASS / [ ] FAIL
3.8 POS reconciliation	[ ] PASS / [ ] FAIL (expected fail)	Confirms Critical Bug 1
3.9 Resume failed audit	[ ] PASS / [ ] FAIL

Overall outcome: [ ] GREEN / [ ] AMBER / [ ] RED

Bugs confirmed during testing:

Critical Bug 1 — pOSOrder cross-service drift
Critical Bug 2 — Audit stuck in RUNNING after side-effect failure
Critical Bug 3 — advanceBusinessDate overwrites settings
Concern #6 — POS reconciliation cross-property leakage
Concern #7 — Folio totals not decremented on rollback

Hand the completed table back to the production launch owner. They will decide whether the bugs block the 2026-04-14 launch.

7. Quick Reference — Service Endpoints

Endpoint	Method	Purpose
`/api/night-audit/:propertyId/start`	POST	Run audit for one property
`/api/night-audit/:propertyId/status`	GET	Get current audit status
`/api/night-audit/:propertyId/history`	GET	Audit history (default 30)
`/api/night-audit/:propertyId/pre-audit-report`	GET	Pre-audit warnings
`/api/night-audit/:propertyId/validate`	GET	Pre-audit validation checks
`/api/night-audit/chain`	POST	Run for all tenant properties
`/api/night-audit/:id/resume`	POST	Resume FAILED audit
`/api/night-audit/:id/rollback`	POST	Rollback COMPLETED audit
`/api/night-audit/record/:id`	GET	Get a single audit record
`/api/night-audit/status`	GET	Status using `x-property-id` header

All endpoints require Authorization: Bearer <JWT> and x-tenant-id header. Most also require x-property-id. Roles required for write operations: SUPER_ADMIN, ADMIN, MANAGER, or ACCOUNTS.

1. Pre-Flight Checklist​

1.1 Connect to QA host​

1.2 Verify all 13 backend services are running​

1.3 Verify the 5-property seed is loaded​

1.4 Verify there is at least 1 in-house reservation per property​

1.5 Confirm no audit is currently running (Redis lock check)​

1.6 Capture a baseline snapshot​

2. Authentication​

3. Test Execution​

Test 3.1 — Pre-audit validation (Scenario 13)​

Test 3.2 — Single-property audit (Scenario 1)​

Test 3.3 — Verify Mumbai audit committed correctly​

Test 3.4 — Verify other properties were untouched (Isolation Test)​

Test 3.5 — Idempotency (Scenario 4)​

Test 3.6 — Chain audit across all 5 properties (Scenario 2)​

Test 3.7 — Cross-property folio leakage check (Concern #6/#7)​

Test 3.8 — POS reconciliation safeguard (Scenario 6)​

Test 3.9 — Resume failed audit (Scenario 9)​

4. Rollback Procedure (If Audit Goes Wrong)​

4.1 Rollback a single audit​

4.2 Hard reset (last resort — destroys all today's audit data)​

4.3 Restore property settings (recover from Critical Bug 3)​

5. Where to Find Logs​

Common log greps​

6. Sign-off Checklist​

7. Quick Reference — Service Endpoints​