Distilling the essence of what the old defaultInfrastructureStackPipeline was actually doing, working chunk by chunk through a real deployment YAML, so we can rebuild it on Wanderland Core for Advaita.
Building Advaita's AWS accounts from scratch. Order of work runs VPC → networking → security groups → load balancers → ECS clusters → ECS services. Every one of those is an infrastructure stack deployment. Nailing the archetype for "stand up a CloudFormation-backed piece of infrastructure" is the prerequisite for everything else in the SOW.
The old defaultInfrastructureStackPipeline archetype had three stages: initialize → deploy → publish. The deploy stage's worker was the infrastructureStack deployer, which fanned out to CreateStack → PublishStack → ActivateStack per declared block. The YAML is what a project team wrote to feed that archetype.
initialze :: required under the Peregrine model for initializing the context of the run. loads environment from existing context, assigns RELEASE | performs query for promotable image
deploy :: creates CF stack, records resources in context for external querys via !Value / internal queries via !Resource and !ComponentResource
pqis-app-deployment/load-balancers/jenkins/preprod-qa.yml---
pipeline:
metadata:
# deprecated
group: pqis
# this becomes the :to_addr value
name: pqis-pqp-pqisappsvc-loadbalancer-preprod-main-qa-pipeline
# this is the system and target environment block, not really just "tags"
tags:
environment: preprod
sub_environment: main
system: pqp.pqisappsvc.loadbalancer
version: qa
# this stamps a new version for integration / selects the image and exports version metadata on promote
version:
strategy: static
args:
release: 1.0.0
stages:
# the publish activities rolled directly into deploy as no validate stage was required to separated them
deploy:
label: deploy-ec2
deployer:
type: infrastructureStack
# components produce resources
components:
# component namespace
default:
# iac provider
cloudformation:
# iac template
template: cloudformation/infrastructure/common_http_https_alb.yml
# template parameters
parameters:
ALBScheme: internal
AvailabilityZoneA: us-east-1a
AvailabilityZoneB: us-east-1c
AvailabilityZoneC: us-east-1f
Environment: !Mapped pipeline.metadata.tags.environment
IdleTimeout: "300"
pipeline.name, group, tags)| key | description | need |
|---|---|---|
| group | original ownership | |
| name | pipeline name, system/env/subenv | X |
| tags | system/env/subenv/version | X |
version is a strategy assignment. during integration, stamps a new version on the artifact - has multiple strategies for external controlled versioning or an automatic escape hatch for (YYYY.MM.BB)
for promotions, uses the image metadata api/db to search for the image corresponding to system + source env and returns the image + metadata to fetch RELEASE
The whole pipeline is one named stage with one worker (infrastructureStack). What the agent label is doing. What the worker fans out to under the hood.
{dive into the actor/activities and document here}
feeds to nimbus create_stack_v2 -> creates/updates cf stack using deterministic naming (system/env/subenv/version), exists = update, not-exists = create. resources captured to pipeline contet
system/env/subenv are lookup key to larger tags structure. look in pqis-app-deployment/tags for yaml files with the config. tagging taxonomy is an open question, at a minimum we need system identity + deployment target identity
we had email/teams, we'll need the same + slack. look in the notifications grovy class,there's a whole dynamic reporting solution in there
| # | Capability | What it has to do | Drives | Case | ALB |
|---|---|---|---|---|---|
| 1 | Run-keyed context store | Each run generates its own context file (or files) at a known location, addressable by run ID. Resources, artifacts, env bindings, stack outputs all written under that ID. | Audit trail; replaces parameter-store-as-context; underpins everything below | ✓ | |
| 2 | Symbolic tags / pointers | Tags or symbolics point to a run (or a group of context files) — same semantics as git tags pointing to a hash. Env-active tag, release tag. | Promotion = move pointer; "what's in prod?" → read pointer | ||
| 3 | Tags as YAML static API via Wanderland Core | YAML file (or git-repo mount) holds the tag taxonomy. A Wanderland Core boundary loads it; a simple chain inside Lantern serves it as an HTTP API. Same pattern as smolchalk's $ref/types.yaml. |
Replaces PQSR; consistent tag merge at deploy time; doc export from the same source | ✓ | |
| 4 | Promotion as pointer-move | Promote = update env tag to point to a different run; no DB lookup, no re-deploy. | Same bytes preprod → prod; auditable; matches peregrine v3 framing | ||
| 5 | Stack output capture | Each CFN output emitted as a crossing, addressable so other runs can read it. | ALB → ECS cross-run handoff; falls out of (1) + (2) | ✓ | |
| 6 | Slack + email notifications | Both channels supported; selectable per result and per environment. | Operator awareness; release coordination; scan-result distribution | ||
| 7 | Composable dynamic reports | Components fetch their data and emit a slice; reports compose components. | Different URLs/info for feature vs prod releases; embed scan results, build URLs, app URLs. Build later — mention now | ||
| 8 | Sibling cleanup | When a new run is active for (system, env, sub_env), prior runs for that key get cleaned up. |
CD environments don't stack dead infrastructure. May fall out of context management — when no tag points to a context, it's a cleanup candidate. | ||
| 9 | TTL expiry | Feature/team contexts get an expiry; expired runs get torn down. | Don't pay for forgotten branches. Context carries expiry as a field. | ||
| 10 | Demolition validation | Versioned set of safety checks ("is anything still hitting this?", traffic checks, dependency checks) runs before tear-down. | Safe destruction even when context-management says "drop it." Keep the validation-rules idea from the old library; drop the two-phase operator-key dance. | ||
| 11 | Provision infrastructure + lifecycle maintenance (CFN-first, Terraform-shaped) | Provision. Push a (template, parameters, capabilities, tags) bundle to a provider, wait on completion, surface outputs as crossings. CFN backend first; same boundary contract fits Terraform later. Capabilities auto-detected from the template — parse for IAM-creating resources, set CAPABILITY_IAM / CAPABILITY_NAMED_IAM without operator action. Tag merge excludes active and lifecycle_state (the old EXCLUDE_TAGS_FROM_UPDATE pattern) so infrastructure updates never trip lifecycle. Lifecycle maintenance. Synchronous tag-flip through the provider. States: new → staged → active → draining → retired → demolished. Invariant: only one run is active per (system, env, sub_env) — activating a new run drains the old in the same call. Idempotency-guarded, optional drift gate. Only active/lifecycle_state tags written; central state in our crossings is authoritative, provider tag follows synchronously. Synchronous, no reconciler. Central state and provider tag stay in sync via same-call writes. Reconciler is a deferred future add for when the stack count makes synchronous writes a coordination problem. CFN's missing query API was the old constraint that drove the periodic-fetch + central-cache pattern; not solving for it today. Rate-aware so a pipeline orchestrating many stacks doesn't trip throttling. |
Archetype core. Without provision there's nothing to capture context about; without lifecycle the active-pointer mechanism (row 2) has nothing on the provider side to read. Maps to old nimbus create_stack_v2 (provision) + StackV2#stage/activate/drain/retire/deactivate (lifecycle). |
✓ | |
| 12 | Artifact registry (system-scoped, dev-hosted) | First-class addressable primitive at :advaita:<system>:dev:artifacts:<artifact_id>. Each artifact carries manifest (image digest / AMI ID / package SHA — references the durable store like ECR), source provenance (git repo, commit, build commands), version field, and from-run back-pointer to the build run that produced it. preprod/prod don't host artifacts — runs in those slots reference dev's via claims-artifact. Multiple runs across env-slots can claim the same artifact (the artifact-reuse / promotion-without-rebuild win). Per-slot pointer tags on the artifact (dev-current, preprod-current, prod-current) kept in sync by the lifecycle boundary's same-call writes. |
Phase 4's "version walks build → prod" — the artifact identity is what flows across runs. Required by row 4 (promotion); fed by build pipelines; cleanup cadence (rows 8/9/10) will eventually walk artifact reference counts to flag orphans. |
Cases get created as we go — one per row, capturing the design decision and the shape of the implementation.
The old code has three distinct cleanup patterns plus a validation layer. Keeping the intent, not the shape.
| Pattern | Old impl | Keep? | New shape |
|---|---|---|---|
| Sibling cleanup | nimbus cleanup --stack_name X — finds DynamoDB deployments matching (system, env, sub_env) and removes the non-active siblings. |
Yes | Falls out of context management. When a new run becomes the env-active pointer, prior runs for the same key are cleanup candidates. |
| TTL expiry | nimbus set_expiration writes a TTL; nimbus expire_stacks polls and deletes by age. Used for feature/team builds. |
Yes | TTL field on the context record; sweeper boundary runs on a schedule. |
| Lifecycle progression | stage / activate / deactivate / drain / retire state machine in DynamoDB; monitor_draining_stacks auto-progresses on traffic dropping below threshold. |
Open question | May fall out of context management. Only matters if we roll stacks rather than update in place. |
| Demolition validation | nimbus controlled demolition — runs a versioned demolition_validations.yml of checks (traffic, dependencies) before allowing destruction. Two-phase: dry-run returns per-stack demolition_key; actual call requires the key. |
Yes (idea), no (UX) | Keep the validation-rules primitive — a set of checks that gate any teardown. Drop the operator-key dance; let the substrate authorize via identity + capability. |
The Ruby helper for controlled demolition is Deployoryah::Jenkins::BobTheDestroyer. We want to maintain the functionality without the 2x UX. Ideally a slack notification - we'd started building teams cards with activity callbacks
| Question | Why open | Unblocks when |
|---|---|---|
| Stack name vs pipeline name — keep two identities or unify | Today nimbus name_from_tags produces a stack name; pipeline name is a separate string. Two identities work; one might be cleaner. |
When we settle context-store addressing. |
| Lifecycle states required, or do they fall out of context management? | Old library uses active / staged / draining / retired for traffic-aware tear-down. Context-management may be enough; depends on whether we ever roll stacks vs update in place. |
First deploy that actually needs traffic-draining. |
| Looping constructs in grids | Not needed today. | A real pipeline forces the question. |
| Multiple things deployed at once (parallel cells) | Not needed today. | A real pipeline forces the question. |
| Archetype hosting — wanderland-core gem, configurable path, or data-driven from config.yml | wanderland-core ships archetypes inside the gem. We may need to extend or work around. | When we write the first archetype. |
| Validation-rules format — versioned YAML or crossings | Demolition rules need to be versioned and queryable; the choice is shape, not function. | Defer until cleanup primitive design. |
The old naming convention - on bu / system / environment / sub_environment / release
nimbus create_stack as a wanderland-core service is a chain — validate inputs → render template → submit CFN call → publish outputs as crossings.