NGA-5114 — Migration Notes

S3 ap-southeast-1 → us-east-1 · curated detail behind the dashboard

Dashboard category meanings

CardWhat it means
Final doneOriginal us-east-1 name live + data restored. Migration complete.
In progressStill mid-flight — staging exists, SG not yet cut over, or data still syncing/draining.
Otherus-east-1-only buckets with no ap-southeast-1 source (orphan/native), that are in the bucket mapping.
Not in scopeSG-only test/infra buckets with no us-east-1 counterpart — intentionally excluded.
Not in bucket MappingTracked buckets whose stem is absent from the Confluence "Bucket Mapping" table (matched by normalized stem).

Not in bucket Mapping — the 13 buckets

App buckets the Confluence plan is missing (real gaps — have staging, being migrated):

Infra / non-app buckets (never part of the plan — done or out of band):

Confluence "Bucket Mapping" — naming problems found

RowIssue
59Typo in us-east-1 name: ncom-… — missing the g, should be ngcom-…
59,60 vs 61,62Dev uses id-ev-hist, perf uses id-event-hist — inconsistent abbreviation
60 vs 62Dev source says -canonical, perf source says -can — inconsistent suffix
59,60Dev comidentitycs raw/can: neither source nor staging bucket exists live — looks stale

Active blockers

ItemBlocker
nex-global-perf-eventhistory-eventstoreDONE (2026-06-30) Full de-import executed: comidentitycs de-import deploy released the export lock → eventhistory renamed n-nex-…-eventstorenex-global-perf-eventhistory-eventstore → 3 metadata objs restored from temp → importValue re-added + redeployed. Trust stayed us-east-1 throughout (no manual patch). Orphan n-nex-… bucket retained for cleanup.
comidentitycs perf OIDCRESOLVED The "trust reverts to ap-southeast-1" was a stale @nextgen-sretool/ams-config pin (resolutions forced 0.6.125, which returns ap-southeast-1 OIDC). Bumped to 0.6.128 (ships us-east-1 OIDC for global-perf) → every comidentitycs deploy now keeps trust on us-east-1 3D01A63F…. No platform-team action needed.
ngcom-global-perf-comidentitycs-identity-event-history-canDONE (2026-07-01) Deployed with skipIAMRole=false skipKafkaConnectors=false (verified against source: both flags gate real constructs — false means "don't skip," keeping the managed SA role and the live Kafka connector intact). Diff showed only the bucket rename + a benign Kafka-connector timestamp re-trigger (self-updating property, fires on every deploy regardless of other changes). Deploy succeeded, no PermanentRedirect. Verified live: bucket exists empty under original name in us-east-1, Kafka connector custom resource UPDATE_COMPLETE (not destroyed).
ngcom-global-perf-comidentitycs-identity-event-history-rawDONE (2026-07-02) Deployed only ~2.5h after SG delete (no PermanentRedirect issue this time). Deploy took ~36 min (2152s) — not stuck, the old us-east-1 staging bucket (id-event-hist-raw) still held ~596,209 objects/versions/markers (~320MB, data pre-confirmed ignore/reproducible), which CDK's AutoDeleteObjectsCustomResource Lambda had to purge before the stack could finish; user accelerated it via the AWS Console's "Empty bucket" action in parallel. Verified live: new bucket exists empty under original name in us-east-1. THIS WAS THE LAST BUCKET — NGA-5114 S3 migration bucket-side work is now COMPLETE.
commerce-document-dataservice — feature branch not pushedPENDING Commit ready locally on feature/NGA-5114-us-east-1 but push fails — account tha05drn_amway has no write access to AmwayCommon/commerce-document-dataservice. Ask AmwayCommon org admin to grant write permission, then run: git push origin feature/NGA-5114-us-east-1 from repos/commerce-document-dataservice.
commerce-order-eventservice — feature branch not pushedPENDING 2 commits ready locally on feature/NGA-5114-us-east-1 (yarn tooling bump + .yarnrc.yml correctly excluded) but push fails — account tha05drn_amway has no write access to AmwayCommon/commerce-order-eventservice (same 403 via terminal git push and GitHub Desktop). Ask AmwayCommon org admin to grant write permission, then git push origin feature/NGA-5114-us-east-1 from repos/commerce-order-eventservice.
All 3 comorderes perf buckets (order-return-latest-event, order-return-events, payment-events)DONE (2026-07-01) cdk deploy succeeded (no PermanentRedirect issue), all 3 live under original names in us-east-1. order-return-events restored from tempmig-bucket-use1/ — verified exact match (42,923 objects / 1,303,580,632 bytes). payment-events already receiving live traffic post-cutover. order-return-latest-event confirmed empty as expected. commerce-order-eventservice repo is fully migrated on the bucket side (push to origin still blocked — see row below).

Draining (data confirmed ignore) — lifecycle emptying SG

Bucket (ap-southeast-1)Objects (versions+markers)SizeTrend (Jun 25 → 28)State
ngcom-global-perf-comorderes-order-return-events58.8M855 GB1.35 TB → 855 GB↓ draining
ngcom-global-perf-comorderes-order-return-latest-event31.8M486 GB1.08 TB → 486 GB↓ draining
ngcom-global-perf-comorderes-payment-events20.5M41.1 GB88.7 GB → 41.1 GB↓ draining
ngcom-global-perf-comidentitycs-identity-event-history-raw0 current1.04 GB76.3 GB → 1.04 GB (Jun 28→29)↓ draining — connector stopped 2026-06-30

Counts as of 2026-06-29 from CloudWatch (NumberOfObjects AllStorageTypes = current + noncurrent versions + delete markers; BucketSizeBytes StandardStorage = real data). The 3 comorderes buckets are genuinely shrinking (real bytes falling) via lifecycle — let them finish. identity-raw is now also draining (2026-06-30): its orphan rawevents3sink connector stopped, so lifecycle collapsed it 76.3 GB → 1.04 GB and it's finishing like the rest.

All four have lifecycle rules: expire current (Day 1) + delete noncurrent versions (Day 1) + remove expired delete markers. Note: NumberOfObjects can look misleading mid-drain — delete markers inflate it even as real bytes fall; judge progress by BucketSizeBytes, not object count. Jump-VM accelerator scripts/delete/drain-bucket-versions.sh (nohup, instance role) deletes versions+markers in 1000-key batches; it was killed 2026-06-29 since it can't win against identity-raw live writes — comorderes lifecycle finishes on its own. After empty → delete SG → Phase C (one comorderes perf deploy strips all 3 aliases).

Remaining buckets (all perf) — 5 left, as of 2026-06-30 (#6 #7 done)

#BucketRepoStatus
1ngcom-global-perf-comorderes-order-return-eventscommerce-order-eventservicedraining 855 GB ↓ — lifecycle
2ngcom-global-perf-comorderes-order-return-latest-eventcommerce-order-eventservicedraining 486 GB ↓ — lifecycle
3ngcom-global-perf-comorderes-payment-eventscommerce-order-eventservicedraining 41 GB ↓ — lifecycle
4ngcom-global-perf-comidentitycs-identity-event-history-rawcommerce-identity-consumerservicedraining 1.04 GB ↓ — connector stopped 2026-06-30, lifecycle finishing
5ngcom-global-perf-comidentitycs-identity-event-history-cancommerce-identity-consumerserviceIN PROGRESS OIDC resolved (ams-config 0.6.128). Statefull-stack bucket rename still pending.
6nex-global-perf-eventhistory-eventstorenextgen-eventhistory-appserviceDONE (2026-06-30) de-import complete — renamed to original, 3 objs restored, importValue reconnected
7nex-global-perf-eventhistory-athena-resultsnextgen-eventhistory-appserviceDONE (2026-06-30) renamed to original (transient, recreated empty)

De-import done: #6 + #7 ✅ (OIDC was a stale ams-config pin, fixed by bump to 0.6.128 — no platform-team dependency). Remaining: #1-4 all draining via lifecycle (#4 connector stopped 2026-06-30 → no longer blocked); #5 Statefull-stack rename. Orphaned n-nex-…-eventstore + n-nex-…-athena-results staging buckets deleted 2026-06-30 ✅.

Pending deploy (user-run)

ItemStatus
nex-global-perf-eventhistory-athena-resultsREADY Edit committed locally on feature/NGA-5114-us-east-1 (eventstore pinned to n- so its export is untouched; athena returns to original name). Vetted via cdk synth. Deploy must run in a normal terminal — the sandbox node errors on cdk deploy. Then scripts/restore/restore-eventhistory-athena-perf.sh.

Repo changes per feature branch (feature/NGA-5114-us-east-1)

RepoFile changedWhat changedPushed
commerce-payment-appservicesrc/modules/s3bucket.tsRemoved n- prefix from bucket name and return value (2 places)
commerce-search-dataservicesrc/modules/s3.tsRemoved n- prefix from bucket name (1 place)
commerce-search-appservicesrc/modules/s3.tsRemoved n- prefix from bucket name (1 place)
commerce-thinsurance-dataservicesrc/modules/s3.tsRemoved n- prefix from bucket name + ARN string
commerce-thlabelprinting-dataservicesrc/modules/s3.tsRemoved n- prefix from bucket name + ARN string
commerce-thlsp-dataservicesrc/modules/s3.tsRemoved n- prefix from bucket name + ARN string; downgraded ams-config 0.6.128 → 0.6.125
nextgen-segments-dataservicesrc/modules/s3Bucket.tsRemoved n- prefix from bucket name and return value (2 places); added packageManager: yarn@4.6.0
commerce-deployment-appservicesrc/modules/s3bucket.tsRemoved n- prefix from bucket name and return value (2 places); downgraded ams-config 0.6.128 → 0.6.125
commerce-order-eventserviceconfigs/global/dev.yamlRemoved -alias suffix from 3 bucket suffixes: order-return-latest-event, order-return-events, payment-events
commerce-identity-migrationservicepackage.jsonPackage version update only (no bucket rename needed)
nextgen-eventhistory-appservicesrc/ (Anish's commit)Removed n- prefix from both athena-results and eventstore bucket names
commerce-document-dataservicestateless_stack.ts, common.yaml, package.jsonCommitted locally — push blocked (no write access to AmwayCommon repo)❌ pending

Key operational rules

Notes maintained alongside the dashboard · NGA-5114