AWS Control Tower Greenfield Runbook — Advaita

Status: Draft — awaiting review.
Engagement:
Owner: Graeme Fawcett

Advaita AWS account structure, identity model, network shape, and provisioning sequence. Account details (root emails, OU placement, CIDRs, configurations) live in their respective sections. Stack details live in the CloudFormation conventions section.

Outstanding Questions

[14/05/26] :: Confirm dev/qa "public" properties should lock down to a) vpn b) ???

Implementation status

CloudFormation templates and parameter files exist in this repo at cloudformation/ on branch aws-raze-and-restore. This runbook describes the target state in prose; the templates encode it as deployable CloudFormation. A reviewer reading the templates should be able to produce a plain-English summary that matches the corresponding section here — discrepancy = bug somewhere, fix both ends.

advaita-deploy/cloudformation/
├── README.md                          Deploy quickref
├── bin/deploy.sh                      ./bin/deploy.sh <system> [<env>] <stack-id>
├── org/                               mgmt-account templates (SCPs, permission sets, baseline StackSets, tag policy)
├── logarchive/                        log-archive-account log buckets
├── audit/                             audit-account aggregator + optional guardduty/inspector/security-hub
└── workload/                          shared templates, per-env params
    ├── templates/                     00-network, 01-bastion, 02-ecr, 03-route53, 04-ecs-foundations, 05-acm
    └── params/{nonprod,prod}/         one params file per template per env

Workload templates are intentionally shared between nonprod and prod — all environment differences (CIDR, NAT count, zone list, scanning mode) live in the per-env parameter files. Adding a third workload env is "create workload/params/<name>/" plus an account.

Design constraints

Constraint	Decision	Notes
Identity provider	Google Workspace via SAML to AWS IAM Identity Center	existing IDP
Compute placement	Private subnets only. Public subnets host ALB/NLB and one EIC-fronted bastion per workload account.	no compute directly exposed, ideally compute (private) > lb (public + restricted) > cdn
Routine instance access	SSM Session Manager	ec2 instance connect, access controlled w/ tags + identity
Break-glass instance access	EC2 Instance Connect Endpoint into the VPC	2am downtime insurance
SSH keys	Forbidden by SCP — no instance launches with a key pair, no inbound `0.0.0.0/0:22`	containerized workloads don't need ssh, internal only SSH on need basis
Governed regions	us-west-2 (primary), us-east-1 (global services only)	existing, minimizes cost overruns in unmonitored regions
DNS apex	`advaitabio.com` on Cloudflare. Per-environment subzones delegated into Route53 in each workload account.	cdn + dns in cloudflare, r53 zones private + origin
IaC	CloudFormation YAML, `AWS::LanguageExtensions` transform enabled	version control all of the things
Stack coupling	Parameter-driven inputs. Outputs published to SSM Parameter Store under `/advaita/<account>/<system>/<resource>`. No `Fn::ImportValue` chains.	existing platform functionality, single source of truth
Mandatory security services	CloudTrail org trail, AWS Config (periodic mode), IAM Access Analyzer	as per best practices
Optional security services	GuardDuty foundational, Inspector ECR Enhanced, Security Hub — enable procedures documented, deferred to a later phase	on a cost/need basis

Account inventory

#	Account name	Root email	OU	Purpose
1	`advaita-management`	`aws-management@advaitabio`	Root	Organizations, Control Tower, IAM Identity Center, billing
2	`advaita-logarchive`	`aws-logarchive@advaitabio`	`Security/`	Centralized CloudTrail + Config log storage
3	`advaita-audit`	`aws-audit@advaitabio`	`Security/`	Security tooling delegated admin
4	`advaita-workload-nonprod`	`aws-nonprod@advaitabio`	`Workloads/Nonprod/`	dev + qa ECS workloads
5	`advaita-workload-prod`	`aws-prod@advaitabio`	`Workloads/Prod/`	prod ECS workloads
6	`advaita-legacy` (existing 789905347053)	`cloud.manager@advaitabio`	`Sandbox/Legacy/`	Existing sprawl; slated for eventual closure

Alternate contacts applied to every account during Account Factory vend:

Role	Email
Billing	`aws-billing@advaitabio`
Security	`aws-security@advaitabio`
Operations	`aws-operations@advaitabio`

All eight aliases created in Google Workspace before any AWS work begins.

Google groups (drive IdC assignments)

Group	Permission Set	Accounts
`aws-org-admins`	`OrgAdmin` (AdministratorAccess)	`advaita-management` only
`aws-platform-admins`	`PlatformAdmin` (AdministratorAccess)	All workload + security accounts
`aws-developers`	`Developer`	`advaita-workload-nonprod` (RW), `advaita-workload-prod` (RO)
`aws-security-readers`	`SecurityAuditor`	`advaita-audit` (RO)
`aws-billing-viewers`	`BillingViewer`	`advaita-management` only
`aws-breakglass`	`BreakGlassAdmin`	All accounts. Empty by default. Granted in writing, revoked after incident.

External CIDR allocations

Workload VPCs land outside everything the rest of the Advaita estate already owns. CoreWeave's two clusters share an identical CIDR plan (documented per-cluster), so AWS dodges all of these ranges:

Range	Span	Owner
`10.0.0.0/13`	`10.0.0.0` – `10.7.255.255`	CoreWeave pod CIDR (per cluster)
`10.16.0.0/22`	`10.16.0.0` – `10.16.3.255`	CoreWeave service CIDR
`10.16.4.0/22`	`10.16.4.0` – `10.16.7.255`	CoreWeave internal LB CIDR
`10.221.128.0/18`	`10.221.128.0` – `10.221.191.255`	CoreWeave host network

AWS workload account allocations:

Account	VPC CIDR	Spans
`advaita-workload-nonprod`	`10.30.0.0/16`	`10.30.0.0` – `10.30.255.255`
`advaita-workload-prod`	`10.40.0.0/16`	`10.40.0.0` – `10.40.255.255`
reserved (future workload)	`10.50.0.0/16`	next

The +10 stride keeps a /16 buffer between accounts so future sub-allocations (extra AZs, secondary CIDRs, peered services) have room. CoreWeave's 10.221 host range is far enough away that the stride hits no collisions before 10.220.

Legacy account (existing 789905347053) is not in this plan — intentionally never peered, eventually closed.

Pre-flight checklist

One-time setup, before AWS provisioning begins. ~1 hour wall time

Google Workspace admin available for SAML app + SCIM token creation (Josh)
Five root-email aliases created (aws-management, aws-logarchive, aws-audit, aws-nonprod, aws-prod) (Josh)
Three alternate-contact aliases created (aws-billing, aws-security, aws-operations) (Josh)
Six Google groups created (aws-org-admins, aws-platform-admins, aws-developers, aws-security-readers, aws-billing-viewers, aws-breakglass) (Josh)
Cloudflare zone for advaitabio.com live; access confirmed to add NS records for subdomain delegation (Graeme/Josh)
AWS support plan decided — Business tier ($100/mo min) is the practical floor for Control Tower (Radu)
Hardware MFA key procured (FIDO2 / YubiKey) for management account root (Radu)
Stakeholder sign-off recorded: 789905347053 will be invited into the new org under Sandbox/Legacy/ (Radu)
DNS migration window agreed for apex advaitabio.com GoDaddy → Cloudflare cutover (Radu/Josh/Graeme/Mike)

1. Management account — `advaita-management`

Foundation account. No workloads run here ever. Contains the organization, Control Tower landing zone, IAM Identity Center, billing, and the operator-facing CloudFormation StackSets.

Identity

Field	Value
Account name	`advaita-management`
Root email	`aws-management@advaitabio`
Alternate billing	`aws-billing@advaitabio`
Alternate security	`aws-security@advaitabio`
Alternate operations	`aws-operations@advaitabio`
Home region	`us-west-2`
Governed regions	`us-west-2`, `us-east-1`

Provisioning (ClickOps)

Sign up new AWS account at aws.amazon.com/free with root email aws-management@advaitabio. Hardware MFA registered immediately, root credentials sealed.
Upgrade support plan to Business.
Enable AWS Organizations with all features. Org ID recorded here once known.
Deploy Control Tower landing zone:- Home region: us-west-2
Governed regions: us-west-2, us-east-1
Log Archive account: create as advaita-logarchive (Section 2 details)
Audit account: create as advaita-audit (Section 3 details)
KMS key for Control Tower: customer-managed, multi-region
Apply OU layout:- Security/ (Log Archive, Audit)
Workloads/Nonprod/
Workloads/Prod/
Sandbox/Legacy/
Suspended/ (empty placeholder for future use)
Enable IAM Identity Center as an organization instance.

SCPs (deployed by CloudFormation StackSet `advaita-org-scps`)

SCP	Target OU	Effect
`deny-region`	Root	Deny all `aws:RequestedRegion` outside `us-west-2`, `us-east-1`. Whitelist for global services (IAM, Route53, CloudFront, etc.).
`deny-root-user`	Root	Deny all action where `aws:PrincipalType == Root`.
`deny-iam-user-create`	`Workloads/`, `Sandbox/`	Block IAM user creation. Forces IdC.
`deny-security-tamper`	Root	Deny disabling CloudTrail, Config, GuardDuty, Security Hub.
`deny-public-ssh`	`Workloads/`	Deny `ec2:AuthorizeSecurityGroupIngress` if port 22 and CIDR `0.0.0.0/0`.
`deny-internet-egress-default`	`Workloads/Prod/`	Belt-and-braces — deny attaching default route to IGW from prod workload (override per-stack).
`require-imdsv2`	`Workloads/`	Deny `ec2:RunInstances` without IMDSv2 required.

Delegated administrators

Service	Delegated to (Account)
CloudTrail org trail	`advaita-logarchive`
Config aggregator	`advaita-audit`
GuardDuty (when enabled)	`advaita-audit`
Security Hub (when enabled)	`advaita-audit`
Inspector (when enabled)	`advaita-audit`
IAM Access Analyzer (org)	`advaita-audit`

IAM Identity Center configuration

External IdP: Google Workspace (Section 7 details).
SCIM provisioning enabled. Token rotated annually.
Six permission sets created (see Section 7).
Account assignments mapped from Google groups to (permission set, account) pairs via CloudFormation cloudformation/org/permission-sets.yaml.

CloudFormation deployed here

Stack	Purpose
`advaita-org-scps`	OU-targeted SCPs (above)
`advaita-org-permission-sets`	IdC permission sets + Google-group account assignments
`advaita-org-stackset-roles`	`AWSCloudFormationStackSetAdministrationRole` + execution role wired into the org
`advaita-org-tag-policy`	Tag policy enforcing `Environment`, `System`, `Owner`, `ManagedBy` tags on tagged resource types
StackSet: `advaita-baseline-config-periodic`	Flip Config to periodic recording in every workload account
StackSet: `advaita-baseline-region-deny`	Defense-in-depth region restriction at account level
StackSet: `advaita-baseline-iam-roles`	`OrganizationAccountAccessRole`, `CFNExecRole`, `BreakGlassAdmin` shells in every member account

2. Log Archive account — `advaita-logarchive`

Sits under Security/. Holds the org-wide CloudTrail trail and Config delivery channel. No humans have write access except via break-glass.

Identity

Field	Value
Account name	`advaita-logarchive`
Root email	`aws-logarchive@advaitabio`
Home region	`us-west-2`

Resources

Resource	Configuration
S3 bucket `advaita-org-cloudtrail-logs`	KMS-CMK encrypted, Object Lock (governance, 7y), block public access, versioning on. Bucket policy allows `cloudtrail.amazonaws.com` writes from org principals only.
S3 bucket `advaita-org-config-logs`	KMS-CMK encrypted, lifecycle to Glacier after 90d, retention 7y. Bucket policy allows `config.amazonaws.com` writes from org principals.
KMS CMK `alias/advaita-logarchive`	Used for both buckets. Key policy grants encrypt/decrypt to org principals for service writes, decrypt to `aws-security-readers` group via IdC role for forensics.
Org trail (`advaita-org-trail`)	Created from `advaita-management`, log target this account's CloudTrail bucket. Management events only (no data events). Multi-region. KMS encrypted.

CloudFormation

Stack	Purpose
`advaita-logarchive-buckets`	Both S3 buckets + KMS key + bucket policies

Outputs (published to SSM in this account)

Path	Value
`/advaita/logarchive/buckets/cloudtrail`	bucket ARN
`/advaita/logarchive/buckets/config`	bucket ARN
`/advaita/logarchive/kms/cmk`	KMS key ARN

3. Audit account — `advaita-audit`

Delegated administrator for org-wide security tooling. Aggregates findings. Read-only for humans.

Identity

Field	Value
Account name	`advaita-audit`
Root email	`aws-audit@advaitabio`
Home region	`us-west-2`

Resources

Resource	Configuration
Config aggregator	Org-wide, aggregates `us-west-2` + `us-east-1` from every member
IAM Access Analyzer	Org-scope analyzer, one per governed region
GuardDuty admin	Off at day-1. Enable procedure documented in Section 10.
Security Hub admin	Off at day-1.
Inspector admin	Off at day-1.
SNS topic `advaita-security-findings`	Sink for high-severity findings once tooling is on

CloudFormation

Stack	Purpose
`advaita-audit-aggregator`	Config aggregator + Access Analyzer
`advaita-audit-guardduty` (optional, deferred)	GuardDuty org enable with foundational sources only
`advaita-audit-inspector` (optional, deferred)	Inspector org enable for ECR enhanced scanning
`advaita-audit-security-hub` (optional, deferred)	Security Hub org enable, Essentials plan

4. Workload — Nonprod (`advaita-workload-nonprod`)

ECS workloads spanning dev + qa, sharing one VPC. Logical separation between the two environments is via private hosted zones, ECS clusters and tag-scoped IAM — not separate VPCs (CIDR conservation, fewer endpoints to pay for).

Identity

Field	Value
Account name	`advaita-workload-nonprod`
Root email	`aws-nonprod@advaitabio`
Home region	`us-west-2`

Network

Item	Value
VPC CIDR	`10.30.0.0/16`
AZs	`us-west-2a`, `us-west-2b`, `us-west-2c`
Public subnets	`10.30.0.0/20` (2a), `10.30.64.0/20` (2b), `10.30.128.0/20` (2c)
Private (compute) subnets	`10.30.16.0/20` (2a), `10.30.80.0/20` (2b), `10.30.144.0/20` (2c)
Intra (no-egress) subnets	`10.30.32.0/20` (2a), `10.30.96.0/20` (2b), `10.30.160.0/20` (2c)
Reserved	`10.30.48.0/20`, `10.30.112.0/20`, `10.30.176.0/20`, `10.30.192.0/18`
Internet gateway	One, attached. Public subnets only.
NAT Gateway	One, AZ-a only (cost-minimal). Private subnets in B/C route cross-AZ.
VPC interface endpoints	`ssm`, `ssmmessages`, `ec2messages`, `ecr.api`, `ecr.dkr`, `logs` — endpoint policy restricts to org principals
VPC gateway endpoint	`s3` (free)
EC2 Instance Connect endpoint	One, in private subnet AZ-a
Bastion	Launch template + ASG `min=0, max=1, desired=0` — t4g.nano AL2023, started on demand, scheduled stop every 4h

DNS

Zone	Type	Notes
`dev.advaitabio.com`	Public Route53	NS-delegated from Cloudflare; ALB aliases for dev services
`qa.advaitabio.com`	Public Route53	NS-delegated from Cloudflare; ALB aliases for qa services
`dev.int.advaitabio.com`	Private Route53	Associated with this VPC; internal-only dev service discovery
`qa.int.advaitabio.com`	Private Route53	Associated with this VPC; internal-only qa service discovery

Compute

ECS clusters: advaita-dev-cluster, advaita-qa-cluster. Fargate only.
ECR repositories: advaita/sc-website, advaita/sc-worker, advaita/az2 (extend via params/nonprod/02-ecr.yaml).
IAM roles: advaita-nonprod-ecs-task-execution shared across clusters; per-service task roles owned by each service's downstream stack.
ALBs: in public subnets, listener :443 only. ACM certs DNS-01 validated against the matching public zone.

CloudFormation stacks (in deploy order)

Stack name	Template	Params
`advaita-nonprod-00-network`	`workload/templates/00-network.yaml`	`workload/params/nonprod/00-network.yaml`
`advaita-nonprod-01-bastion`	`workload/templates/01-bastion.yaml`	`workload/params/nonprod/01-bastion.yaml`
`advaita-nonprod-02-ecr`	`workload/templates/02-ecr.yaml`	`workload/params/nonprod/02-ecr.yaml`
`advaita-nonprod-03-route53`	`workload/templates/03-route53.yaml`	`workload/params/nonprod/03-route53.yaml`
`advaita-nonprod-04-ecs-foundations`	`workload/templates/04-ecs-foundations.yaml`	`workload/params/nonprod/04-ecs-foundations.yaml`
`advaita-nonprod-05-acm`	`workload/templates/05-acm.yaml`	`workload/params/nonprod/05-acm.yaml`

Deploy each: ./bin/deploy.sh workload nonprod <stack-id> (e.g. ./bin/deploy.sh workload nonprod 00-network).

Outputs (published to SSM)

Path	Value
`/advaita/nonprod/network/vpc-id`	VPC ID
`/advaita/nonprod/network/vpc-cidr`	VPC CIDR
`/advaita/nonprod/network/public-subnets`	Comma-list of public subnet IDs
`/advaita/nonprod/network/private-subnets`	Comma-list of private subnet IDs
`/advaita/nonprod/network/intra-subnets`	Comma-list of intra subnet IDs
`/advaita/nonprod/network/eic-endpoint-id`	EIC endpoint ID
`/advaita/nonprod/network/endpoint-sg-id`	VPC endpoints security group ID
`/advaita/nonprod/bastion/asg-name`	Bastion ASG name
`/advaita/nonprod/bastion/sg-id`	Bastion SG ID
`/advaita/nonprod/ecr/advaita/sc-website/uri`	ECR repo URI
`/advaita/nonprod/ecr/advaita/sc-worker/uri`	ECR repo URI
`/advaita/nonprod/ecr/advaita/az2/uri`	ECR repo URI
`/advaita/nonprod/route53/dev.advaitabio.com/zone-id`	Public zone ID
`/advaita/nonprod/route53/dev.advaitabio.com/nameservers`	NS records to paste into Cloudflare apex
`/advaita/nonprod/route53/qa.advaitabio.com/zone-id`	Public zone ID
`/advaita/nonprod/route53/qa.advaitabio.com/nameservers`	NS records to paste into Cloudflare apex
`/advaita/nonprod/route53/dev.int.advaitabio.com/zone-id`	Private zone ID
`/advaita/nonprod/route53/qa.int.advaitabio.com/zone-id`	Private zone ID
`/advaita/nonprod/ecs/task-execution-role-arn`	Shared ECS task-exec role ARN
`/advaita/nonprod/ecs/advaita-dev-cluster/cluster-arn`	Dev cluster ARN
`/advaita/nonprod/ecs/advaita-qa-cluster/cluster-arn`	QA cluster ARN
`/advaita/nonprod/acm/dev.advaitabio.com/cert-arn`	Dev wildcard cert ARN
`/advaita/nonprod/acm/qa.advaitabio.com/cert-arn`	QA wildcard cert ARN

5. Workload — Prod (`advaita-workload-prod`)

Prod ECS workloads. Same template family as nonprod, different params (prod CIDR, three NAT Gateways for AZ resilience, one prod public zone, ECR Enhanced scanning on by default). The Workloads/Prod OU carries an additional SCP layer not applied to nonprod.

Identity

Field	Value
Account name	`advaita-workload-prod`
Root email	`aws-prod@advaitabio`
Home region	`us-west-2`

Network

Item	Value
VPC CIDR	`10.40.0.0/16`
AZs	`us-west-2a`, `us-west-2b`, `us-west-2c`
Public subnets	`10.40.0.0/20`, `10.40.64.0/20`, `10.40.128.0/20`
Private subnets	`10.40.16.0/20`, `10.40.80.0/20`, `10.40.144.0/20`
Intra subnets	`10.40.32.0/20`, `10.40.96.0/20`, `10.40.160.0/20`
NAT Gateways	Three (one per AZ)
VPC interface endpoints	Same set as nonprod, in all three AZs
EIC endpoint	One, AZ-a
Bastion	t4g.nano ASG `min=0,max=1,desired=0`, scheduled stop every 4h

DNS

Zone	Type
`prod.advaitabio.com`	Public Route53, NS-delegated from Cloudflare
`prod.int.advaitabio.com`	Private Route53, associated with VPC

Compute

ECS cluster: advaita-prod-cluster. Fargate only.
ECR: same logical repo names as nonprod; EnhancedScanning: 'true' in params/prod/02-ecr.yaml turns on Inspector-powered continuous CVE scanning.
Lifecycle policy retains the last 100 tagged images per repo.
Prod-OU SCP adds: deny disabling S3 public-access-block; (extendable) deny PassRole to roles not tagged Environment: prod.

CloudFormation stacks (in deploy order)

Same templates as nonprod, prod params:

Stack name	Template	Params
`advaita-prod-00-network`	`workload/templates/00-network.yaml`	`workload/params/prod/00-network.yaml`
`advaita-prod-01-bastion`	`workload/templates/01-bastion.yaml`	`workload/params/prod/01-bastion.yaml`
`advaita-prod-02-ecr`	`workload/templates/02-ecr.yaml`	`workload/params/prod/02-ecr.yaml`
`advaita-prod-03-route53`	`workload/templates/03-route53.yaml`	`workload/params/prod/03-route53.yaml`
`advaita-prod-04-ecs-foundations`	`workload/templates/04-ecs-foundations.yaml`	`workload/params/prod/04-ecs-foundations.yaml`
`advaita-prod-05-acm`	`workload/templates/05-acm.yaml`	`workload/params/prod/05-acm.yaml`

Deploy each: ./bin/deploy.sh workload prod <stack-id>.

Outputs

Same SSM path shape as nonprod with prod as the account segment. Examples: /advaita/prod/network/vpc-id, /advaita/prod/ecs/advaita-prod-cluster/cluster-arn, /advaita/prod/acm/prod.advaitabio.com/cert-arn, /advaita/prod/route53/prod.advaitabio.com/nameservers.

6. Legacy — `advaita-legacy` (789905347053)

The existing pre-Control-Tower workload account. Brought into the new org under Sandbox/Legacy/ to consolidate billing and apply org guardrails. Not networked into nonprod/prod. Slated for closure as workloads migrate to new accounts or to CoreWeave.

Inventory snapshot at intake (2026-05-13)

Class	Count / notes
VPCs	5 in us-west-2; three overlap on `10.0.0.0/16`
ECS clusters	12 (legacy + current — `cdev2-services`, `prod2-services`, `prod-workers`, etc.)
Long-lived EC2	~12 instances (Neo4j fleet, `barad-dur` Eureka+ConfigServer, dockerhub mirror)
IAM users	45 — to be retired as IdC takes over
IAM roles	72 — most retire with the workloads they serve
RDS	`clouddev` Postgres (ars DB)
ECR repos	`az2-cw`, `iscanguide/sc-website`, `iscanguide/sc-worker`
Public Route53 zone	`cw.advaitabio.com.` (re-homed when DNS migrates to Cloudflare apex)

Intake actions

Invite to org via Control Tower → accept from legacy account → land in Sandbox/Legacy/
Apply baseline SCPs from the org (deny region, deny security tamper, require IMDSv2 — IAM user creation is allowed in legacy OU only)
IAM users grandfathered in place. New access granted only via IdC.
Cost tag policy enforced (Environment: legacy, Owner: platform)
No VPC peering, no TGW attachment, no shared services

Retirement criteria (per-workload)

A workload is removed from legacy when:

The new home (CoreWeave or new workload account) is verified end-to-end.
DNS is cutover with monitored TTL.
The legacy stack has been off for ≥30 days without incident.
The IAM users / roles unique to that workload are deleted.

Closure of the account itself

Triggered when the inventory above is zero (or near-zero, cost ≤ $50/mo for residual storage). Standard account-closure process from advaita-management.

7. Identity — Google Workspace ↔ IAM Identity Center

Google Workspace side (ClickOps - Josh)

Admin console → Apps → Web and mobile apps → Add custom SAML app, name AWS IAM Identity Center.
Download Google IdP metadata XML.
Configure SAML attributes:- NameID → email (primary)
email → email
firstName → first_name
lastName → last_name
Enable SAML app for the relevant Google OU.
Create SCIM provisioning token (AWS sets this up; Google admin pastes the SCIM endpoint URL and bearer token AWS produces).
Set group push: push the six aws-* groups to AWS.

AWS IAM Identity Center side (ClickOps - Graeme)

From advaita-management, Identity Center → Settings → Identity source → Change to External identity provider.
Upload Google's IdP metadata XML. Note IdC's ACS URL + audience; paste these back into the Google SAML app.
Settings → Automatic provisioning → Enable. Note SCIM endpoint + token; paste into the Google admin SAML app's SCIM page.
Wait for first sync. Users + groups appear in IdC.
CloudFormation cloudformation/org/permission-sets.yaml creates the six permission sets and maps group → (permission set, account).

Permission sets

Permission set	Policies	Session	Notes
`OrgAdmin`	`AdministratorAccess`	4h	Mgmt account only
`PlatformAdmin`	`AdministratorAccess`	4h	Workload + security accounts
`Developer`	`PowerUserAccess` + denyboundary on IAM + KMS write	8h	Workload-nonprod RW, workload-prod read-only via custom policy
`SecurityAuditor`	`SecurityAudit` + `ViewOnlyAccess`	8h	Audit account
`BillingViewer`	`Billing` + `AWSBillingReadOnlyAccess`	8h	Mgmt only
`BreakGlassAdmin`	`AdministratorAccess`	1h	All accounts. Empty group by default. Grant logged via CloudTrail + Slack alert.

Operator workstation setup

Each operator runs:

aws configure sso
# SSO start URL: https://advaita.awsapps.com/start
# SSO region: us-west-2

Then their ~/.aws/config looks like:

[profile mgmt]
sso_session = advaita
sso_account_id = <mgmt-account-id>
sso_role_name = OrgAdmin
region = us-west-2

[profile nonprod-admin]
sso_session = advaita
sso_account_id = <nonprod-account-id>
sso_role_name = PlatformAdmin
region = us-west-2

[profile prod-admin]
sso_session = advaita
sso_account_id = <prod-account-id>
sso_role_name = PlatformAdmin
region = us-west-2

[sso-session advaita]
sso_start_url = https://advaita.awsapps.com/start
sso_region = us-west-2
sso_registration_scopes = sso:account:access

Auth: aws sso login --sso-session advaita.

8. SSM + EC2 Instance Connect — zero-SSH access pattern

Instance baseline

AMI: Amazon Linux 2023, SSM Agent preinstalled
Instance profile: AmazonSSMManagedInstanceCore (managed policy) + ECR pull for ECS hosts
No key pair on launch (CFN parameter KeyName left empty; SCP would block use anyway)
IMDSv2 required (SCP require-imdsv2)
Public IP: never assigned to compute instances; only the ALB and bastion (when started) live in public subnets
Security group: no inbound except port 22 from the EIC endpoint security group

VPC endpoint set (private DNS enabled, all interface endpoints)

Endpoint	Purpose
`com.amazonaws.<region>.ssm`	SSM control plane
`com.amazonaws.<region>.ssmmessages`	Session Manager data channel
`com.amazonaws.<region>.ec2messages`	SSM Agent → EC2 messaging
`com.amazonaws.<region>.ecr.api`	ECR control plane (for ECS image pulls)
`com.amazonaws.<region>.ecr.dkr`	ECR registry pulls
`com.amazonaws.<region>.logs`	CloudWatch Logs
`com.amazonaws.<region>.s3` (gateway, free)	ECR layer storage, Session Manager logs

Endpoint policy restricts to org principals via aws:PrincipalOrgID.

EIC endpoint

One per workload VPC, in a private subnet
Security group: outbound to instance-side SG on port 22
Instance SG accepts port 22 from the EIC SG only
Operator connects: aws ec2-instance-connect ssh --instance-id i-… --profile nonprod-admin
IAM gate: ec2-instance-connect:OpenTunnel action allowed on the EIC endpoint, scoped to instance tag

Bastion (escape hatch only)

ASG min=0, max=1, desired=0 — bastion lives as a launch template; ASG scales to 1 on demand and back to 0 after 4h via EventBridge schedule
t4g.nano AL2023 with SSM Agent only — no inbound SSH from anywhere; access is via SSM Session Manager into the bastion, then onward connections from there
Cost when stopped: $0 compute, ~$0 EBS for the 8GB root volume
Used for: third-party SaaS that requires a fixed egress IP, ad-hoc network debugging from "inside the VPC"

Operator cheat sheet

# Routine: shell into any ECS host or EC2 by ID
aws ssm start-session --target i-0123456789abcdef0 --profile nonprod-admin

# Break-glass: SSH through EIC endpoint (uses ephemeral key)
aws ec2-instance-connect ssh --instance-id i-0123… --profile nonprod-admin

# Port-forward through SSM (RDS, Elasticache, etc.)
aws ssm start-session --target i-0123… \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["rds.…"],"portNumber":["5432"],"localPortNumber":["5432"]}' \
  --profile nonprod-admin

# Bring up the bastion when needed
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name advaita-nonprod-bastion --desired-capacity 1 \
  --profile nonprod-admin

9. Container scanning

This is a decision that needs to be made /eventually/

Two paths documented. Either is fine. Both used together gives defense in depth.

Path A: Trivy in Jenkins (DIY, $0)

Trivy runs as a build step in every Jenkins pipeline that produces an image
Gates on CRITICAL/HIGH CVEs; configurable severity threshold per repo
SARIF output published to the build artifact
Failure breaks the build before ECR push
Pros: free, build-time gate, results in your CI dashboards, vendor-neutral
Cons: another tool to maintain; depends on Trivy DB update cadence in the build image

Path B: ECR Enhanced Scanning (Inspector, ~$6–15/mo)

ECR repo flag scanningConfiguration: { scanType: ENHANCED }
On push: full SBOM + CVE scan via Inspector
Continuous re-scan: when new CVEs published against existing layers
Findings visible in ECR console + Inspector + (later) Security Hub
Pros: continuous (catches CVEs disclosed after push), no extra plumbing, native integration
Cons: post-push (image already exists when scanned), costs money
Pricing (current): $0.09 per initial scan, $0.01 per rescan. Typical Advaita push volume: ~$6–15/mo across all repos.

Recommendation

Use both. Trivy in Jenkins is the gate (blocks bad images at build time). ECR Enhanced is the safety net (catches the CVE disclosed three weeks after the image was already shipped). Total cost: ~$10/mo for the safety net.

Enable ECR Enhanced via cloudformation/<workload>/02-ecr.yaml parameter EnhancedScanning: true.

10. Security services — cost and posture

Day-1 mandatory (built into Control Tower)

Service	Mode	Estimated cost
CloudTrail org trail (mgmt events)	One free trail per account	$0
AWS Config	Periodic recording, 2 regions	$40–80/mo across 6 accounts
IAM Access Analyzer	Org analyzer in audit	$0
Subtotal		$40–80/mo

Optional, deferred (documented enable procedures)

Service	When to turn on	Estimated cost
GuardDuty (foundational sources only)	Once workload traffic is steady; use 30-day free trial to size	$40–90/mo across 6 accounts
GuardDuty Runtime Monitoring	Only if runtime threat detection on ECS tasks is specifically needed	$1.50/vCPU/mo × ~50 vCPU = +$75/mo
Inspector ECR Enhanced	Recommended day-1 if container scanning is wanted; cheapest line item	$6–15/mo
Inspector EC2 (agent)	If long-lived EC2 remains in legacy account	$1.258/instance/mo × ~12 = ~$15/mo
Security Hub Essentials	After GuardDuty + Inspector are on (has more to aggregate)	$3.75/resource-unit/mo. Size via 30-day unlimited trial.

Enable procedure (per service)

All via CloudFormation in the audit account using delegated-admin org enablement. Steps:

From advaita-management, register advaita-audit as delegated admin for the service (one CLI call: aws securityhub enable-organization-admin-account or equivalent).
Deploy the corresponding stack from cloudformation/audit/ in the audit account.
Stack configures: enable in all member accounts via auto-enrollment, set finding destinations, write SNS event for HIGH+CRITICAL.
Verify in audit account console.

11. ClickOps vs CloudFormation — division of labor

Has to be ClickOps (or so close to it that automation isn't worth it)

Step	Reason
Create new mgmt AWS account	Web signup; no org exists yet to vend from
Enable Organizations + Control Tower landing zone	CT bootstrap is console-driven in practice; one-shot
IAM Identity Center initial enable	Console toggle; ~10 seconds
Google Workspace ↔ IdC SAML metadata exchange	Both ends admin consoles; ~10 minutes
Account Factory vend of LogArchive / Audit / Workload-NP / Workload-Prod	CT console form per account; ~5 minutes each. AFT framework is overkill at 6 accounts.
Invite 789905347053 into Sandbox/Legacy	Console invite + email confirmation
Hardware MFA registration on roots	By definition manual

Everything else is CloudFormation YAML

Per-account stack layout under advaita-deploy/cloudformation/. See Section 12.

12. CloudFormation conventions

Repo layout (actual)

advaita-deploy/cloudformation/
├── README.md                          Deploy quickref
├── bin/deploy.sh                      ./bin/deploy.sh <system> [<env>] <stack-id>
├── bin/verify.sh                      Dry-run + capture resolved-params, resources, outputs
├── config.yaml                        Cross-cutting values (OrgId, account IDs, OU IDs, IdC group IDs)
├── org/
│   ├── templates/
│   │   ├── org-scps.yaml
│   │   ├── permission-sets.yaml
│   │   ├── stackset-roles.yaml
│   │   ├── tag-policy.yaml
│   │   ├── stackset-baseline-config-periodic.yaml
│   │   ├── stackset-baseline-region-deny.yaml
│   │   └── stackset-baseline-iam-roles.yaml
│   └── params/                        (one matching params file per template)
├── logarchive/
│   ├── templates/log-buckets.yaml
│   └── params/log-buckets.yaml
├── audit/
│   ├── templates/
│   │   ├── aggregator.yaml             mandatory
│   │   ├── guardduty.yaml              optional, deferred
│   │   ├── inspector.yaml              optional
│   │   └── security-hub.yaml           optional, deferred
│   └── params/
└── workload/                           shared templates, per-env params
    ├── templates/
    │   ├── 00-network.yaml
    │   ├── 01-bastion.yaml
    │   ├── 02-ecr.yaml
    │   ├── 03-route53.yaml
    │   ├── 04-ecs-foundations.yaml
    │   └── 05-acm.yaml
    └── params/
        ├── nonprod/                    6 yaml files, one per template
        └── prod/                       6 yaml files, one per template

Authoring rules

Transform AWS::LanguageExtensions enabled where loops / JSON conversions are needed (workload templates, org SCPs, ECR repo loops). Use Fn::ForEach for per-AZ / per-list resources, Fn::ToJsonString for embedded policies.
One stack per file. File name = stack id. 00-network.yaml deploys as advaita-<env>-00-network.
Every input is a Parameter. No hardcoded account IDs, CIDRs, names or ARNs in template bodies. Workload templates are shared between nonprod and prod — environment differences live in workload/params/<env>/*.yaml.
Outputs published to SSM Parameter Store. Every resource a downstream stack or operator might reference is written to /advaita/<account-short>/<system>/<resource> via AWS::SSM::Parameter resources inside the producing stack. Downstream stacks consume by passing the parameter name in and resolving via {{resolve:ssm:...}} at deploy time. No Fn::ImportValue chains — producer can be torn down and recreated freely.
Stack outputs also retained as standard CFN Outputs for human inspection (aws cloudformation describe-stacks --query Stacks[0].Outputs).
Tags on every resource: Environment, System, Owner: platform, ManagedBy: cloudformation, Repo: advaita-deploy. Tag policy in advaita-management enforces presence on tagged resource types.
Stack naming: advaita-<account-short>-<stack-id> — e.g. advaita-nonprod-00-network, advaita-prod-02-ecr, advaita-org-org-scps, advaita-logarchive-log-buckets.
Deployments via SSO assumed role. Operator runs aws sso login --sso-session advaita, then export AWS_PROFILE=<target-profile>, then ./bin/deploy.sh ....
**Bucket names suffixed with ${AWS::AccountId}`** — log buckets are constructed as `advaita-org-cloudtrail-${AWS::AccountId} and advaita-org-config-${AWS::AccountId} so the global S3 namespace race is impossible by construction.
Drift detection scheduled monthly via EventBridge → Lambda → SNS in the audit account (not built day-one).
Changes via change sets for prod stacks: aws cloudformation create-change-set → human review → execute. Optional for nonprod.

Pipeline integration (future)

Each (template, params) pair is designed to become a cloudformation_stack_deploy component in a Vivarta pipeline. The pipeline declares named components that reference the same files the operator deploys by hand today, walks them in order, every deployment recorded for audit. Local-CLI deploy stays the same after the pipeline is wired — same script, same files.

See ~/working/vivarta-jenkins/vivarta.yml for the component shape.

13. Provisioning sequence

In order. Each step is independently validated before the next begins. ./bin/deploy.sh and ./bin/verify.sh calls require the appropriate AWS_PROFILE for the target account.

cloudformation/config.yaml holds cross-cutting values (OrgId, account IDs, OU IDs, IdC group IDs). It's filled in incrementally as each provisioning step produces new IDs. verify.sh reads it, substitutes into params, and dry-runs every stack before any deploy.

#	Step	Where	Mode	Command
1	Pre-flight complete — emails, groups, Cloudflare ready	Google Workspace + Cloudflare	ClickOps	—
2	Create new mgmt AWS account; hardware MFA on root; upgrade support to Business	AWS signup	ClickOps	—
3	Enable Organizations all-features; enable Control Tower in us-west-2; add us-east-1 as governed region	mgmt console	ClickOps	—
4	Capture OU IDs + Org ID into `config.yaml`	mgmt	edit	`aws organizations list-roots` + `list-organizational-units-for-parent`
5	Account Factory vend: `advaita-logarchive`, `advaita-audit`	mgmt → CT	ClickOps	—
6	Capture logarchive + audit account IDs into `config.yaml`	mgmt	edit	`aws organizations list-accounts`
7	Verify org / logarchive / audit stacks	local	dry-run	`./bin/verify.sh org && ./bin/verify.sh logarchive && ./bin/verify.sh audit`
8	Deploy log buckets in logarchive account	logarchive	CFN	`./bin/deploy.sh logarchive log-buckets`
9	Deploy aggregator + Access Analyzer in audit account	audit	CFN	`./bin/deploy.sh audit aggregator`
10	Enable IAM Identity Center; configure Google Workspace SAML + SCIM	mgmt console + Google admin	ClickOps	—
11	Capture IdC InstanceArn + Group IDs into `config.yaml`	mgmt	edit	`aws sso-admin list-instances` + `aws identitystore list-groups`
12	Deploy StackSet admin role	mgmt	CFN	`./bin/deploy.sh org stackset-roles`
13	Deploy SCPs (root + workload + prod-OU layers)	mgmt	CFN	`./bin/deploy.sh org org-scps`
14	Deploy permission sets + group assignments	mgmt	CFN	`./bin/deploy.sh org permission-sets`
15	Deploy tag policy	mgmt	CFN	`./bin/deploy.sh org tag-policy`
16	Deploy baseline StackSets	mgmt	CFN	`./bin/deploy.sh org stackset-baseline-config-periodic` → `...-region-deny` → `...-iam-roles`
17	Account Factory vend: `advaita-workload-nonprod`, `advaita-workload-prod`	mgmt → CT	ClickOps	—
18	Capture nonprod + prod account IDs into `config.yaml`	mgmt	edit	`aws organizations list-accounts`
19	Re-deploy permission-sets with full account list	mgmt	CFN	`./bin/deploy.sh org permission-sets`
20	Verify all nonprod stacks; review captured resources	local	dry-run	`./bin/verify.sh workload nonprod`
21	Deploy nonprod workload stacks in order	nonprod (assumed role)	CFN	`./bin/deploy.sh workload nonprod 00-network` → `01-bastion` → `02-ecr` → `03-route53` → `04-ecs-foundations` → `05-acm`
22	Copy nonprod public-zone NS records from SSM to Cloudflare apex	Cloudflare	ClickOps	`aws ssm get-parameter --name /advaita/nonprod/route53/dev.advaitabio.com/nameservers` (repeat for qa)
23	Verify + deploy prod workload stacks in same order	prod (assumed role)	CFN	`./bin/verify.sh workload prod` then `./bin/deploy.sh workload prod 00-network` → `...` → `05-acm`
24	Copy prod public-zone NS records to Cloudflare apex	Cloudflare	ClickOps	—
25	Invite 789905347053 into `Sandbox/Legacy/`; accept from legacy	mgmt + legacy	ClickOps	—
26	Validation: deliberate guardrail violation tests (open `0.0.0.0/0:22`, launch in `eu-west-1`, create an IAM user in nonprod)	nonprod	manual	—
27	Operator handoff: profiles configured, runbook walked through with at least one other engineer	workstation	manual	—
28	Optional: register audit as delegated admin for Inspector; deploy enhanced scanning enablement	mgmt + audit	CLI + CFN	`aws inspector2 enable-delegated-admin-account ...` then `./bin/deploy.sh audit inspector`
29	Optional: register audit as delegated admin for GuardDuty; deploy foundational enablement	mgmt + audit	CLI + CFN	`aws guardduty enable-organization-admin-account ...` then `./bin/deploy.sh audit guardduty`
30	Optional: register audit as delegated admin for Security Hub; deploy on Essentials	mgmt + audit	CLI + CFN	`aws securityhub enable-organization-admin-account ...` then `./bin/deploy.sh audit security-hub`

Both deploy.sh and verify.sh refuse to run if their params file contains unresolved placeholders (any value starting with REPLACE_ME that isn't backed by an entry in config.yaml). Fill config.yaml first, re-verify, then deploy.

14. Future concerns

DNS apex migration (GoDaddy → Cloudflare) — independent of AWS work but blocks public hostname cutover. Cloudflare hosts apex; Route53 hosts the delegated <env>.advaitabio.com subzones in workload accounts. SerNet / China-region implications flagged for separate investigation.
Transit Gateway — intentionally not built day-one. Workload accounts are isolated. Legacy never connects. If a cross-account networking need emerges, TGW in advaita-management (or a dedicated networking account) with attachments per workload VPC is the canonical pattern.
ACM Private CA — internal certs for <env>.int.advaitabio.com need an internal CA if mTLS is wanted. Defer.
CoreWeave OIDC trust — workloads moving to CoreWeave will need IAM Roles via OIDC federation. Pattern documented in [advaita-stuff](advaita-stuff) (under engagement framings). Add when needed.
Account closure of advaita-legacy — gated on workload migration completion (see Section 6 retirement criteria).
CT Customizations Framework (CfCT) — not adopted. Could be reconsidered if account count grows beyond ~10 or if landing-zone customization becomes complex.

AWS Control Tower Greenfield Runbook — Advaita

Outstanding Questions

Implementation status

Design constraints

Account inventory

Google groups (drive IdC assignments)

External CIDR allocations

Pre-flight checklist

1. Management account — advaita-management

Identity

Provisioning (ClickOps)

SCPs (deployed by CloudFormation StackSet advaita-org-scps)

Delegated administrators

IAM Identity Center configuration

CloudFormation deployed here

2. Log Archive account — advaita-logarchive

Identity

Resources

CloudFormation

Outputs (published to SSM in this account)

3. Audit account — advaita-audit

Identity

Resources

CloudFormation

4. Workload — Nonprod (advaita-workload-nonprod)

Identity

Network

DNS

Compute

CloudFormation stacks (in deploy order)

Outputs (published to SSM)

5. Workload — Prod (advaita-workload-prod)

Identity

Network

DNS

Compute

CloudFormation stacks (in deploy order)

Outputs

6. Legacy — advaita-legacy (789905347053)

Inventory snapshot at intake (2026-05-13)

Intake actions

Retirement criteria (per-workload)

Closure of the account itself

7. Identity — Google Workspace ↔ IAM Identity Center

Google Workspace side (ClickOps - Josh)

AWS IAM Identity Center side (ClickOps - Graeme)

Permission sets

Operator workstation setup

8. SSM + EC2 Instance Connect — zero-SSH access pattern

Instance baseline

VPC endpoint set (private DNS enabled, all interface endpoints)

EIC endpoint

Bastion (escape hatch only)

Operator cheat sheet

9. Container scanning

Path A: Trivy in Jenkins (DIY, $0)

Path B: ECR Enhanced Scanning (Inspector, ~$6–15/mo)

Recommendation

10. Security services — cost and posture

Day-1 mandatory (built into Control Tower)

Optional, deferred (documented enable procedures)

Enable procedure (per service)

11. ClickOps vs CloudFormation — division of labor

Has to be ClickOps (or so close to it that automation isn't worth it)

Everything else is CloudFormation YAML

12. CloudFormation conventions

Repo layout (actual)

Authoring rules

Pipeline integration (future)

13. Provisioning sequence

14. Future concerns

1. Management account — `advaita-management`

SCPs (deployed by CloudFormation StackSet `advaita-org-scps`)

2. Log Archive account — `advaita-logarchive`

3. Audit account — `advaita-audit`

4. Workload — Nonprod (`advaita-workload-nonprod`)

5. Workload — Prod (`advaita-workload-prod`)

6. Legacy — `advaita-legacy` (789905347053)