Wanderland

Advaita Product: iSCanGuide

iSCanGuide is Advaita's single-cell analysis web application. The deployment lives in ~/working/advaita-deploy/projects/iscanGuide as a Kustomize tree with three Kubernetes overlays (dev/qa/prod) plus a minikube overlay for local laptop runs, fronted by a pair of bash deployment scripts.

These notes are distilled from the configs on develop as of 2026-05-11. They are a baseline to validate against once read-only access to the running clusters is in hand.

Workloads

The application is four Kubernetes workloads in a single namespace per environment.

Workload Kind Image Port Purpose
sc-website Deployment iscanguide/sc-website (ECR) 3000 Meteor/React web frontend + DDP WebSocket
sc-task-queue Deployment iscanguide/sc-worker (ECR) 9000 (log-server) Node dispatcher bundling conda + R/Seurat scripts
server-zoom Deployment iscanguide/xenium-vis-gpu (ECR) 8080 Xenium zoom/render backend — server-side figure export for iScanGuide
mongodb StatefulSet mongo:7.0.5 27017 Single-replica replica set rs0 — interim, Atlas planned

sc-task-queue is explicitly not horizontally scaled — queue state is in-process, replicas: 1, strategy: Recreate. sc-website defaults to 2 replicas with HPA targeting 70% CPU (min 1, max 3); sticky sessions are required for Meteor's DDP and are pinned by the METEOR_AFFINITY cookie at the nginx ingress.

R/Seurat job scripts are baked into the sc-worker image; there is no scripts PVC.

server-zoom is the Xenium render service behind iScanGuide (added 2026-07). It serves the /api_scatter and /api_spatial figure-export endpoints that sc-website relays to server-side (XENIUM_RENDER_URL=http://server-zoom:8080), reading study .h5 files from the shared data-pvc at /data. It is namespace-internal (ClusterIP, no ingress), CPU-only, replicas: 1, strategy: Recreate. Image iscanguide/xenium-vis-gpu; source repo sc-website-zoom-service.

Storage

All three PVCs use the CoreWeave shared-vast storage class (Vast Data, RWX). Each overlay patches the base sizes:

PVC Base dev qa prod Mounts
data-pvc 1 Ti 500 Gi 1 Ti 10 Ti /data on website + worker
params-pvc 10 Gi 3 Gi 3 Gi 10 Gi (base) /params on website + worker
logs-pvc 50 Gi 8 Gi 16 Gi 50 Gi (base) /logs on worker only

data-pvc in prod was grown to 10 Ti after a qa EDQUOT incident on 2026-05-06. The dev data-pvc patch notes that it was grown in-cluster to 500 Gi and the manifest is tracking observed reality (Kubernetes won't shrink PVCs).

MongoDB's mongo-data is a per-pod 20 Gi ReadWriteOnce volume from volumeClaimTemplates, also on shared-vast.

Environments

Env Namespace Cluster Overlay URL
dev iscanguide-dev useast13-dev k8s/overlays/dev https://iscanguide-dev.cw.advaitabio.com
qa iscanguide-qa useast13-prod k8s/overlays/qa https://iscanguide-bd.advaitabio.com
prod iscanguide useast13-prod k8s/overlays/prod https://iscanguide.advaitabio.com

QA and prod share the same physical cluster (useast13-prod, single 8-GPU node: 8× RTX 6000 Pro, 128 CPU, ~1 TB RAM). Isolation between qa and prod is enforced by namespace + ResourceQuota + NetworkPolicy + PriorityClass — there is no separate cluster for QA. Dev is on its own cluster (useast13-dev).

The minikube overlay is the laptop variant: storageClass standard, shrunk sizes, imagePullPolicy: Never, SUBSCRIPTION_DISABLED=1, reduced worker resources (R jobs do not run). Hostname iscanguide.test:8080.

ResourceQuotas (per namespace)

Quota dev qa prod
cpu (req/lim) 16 / 16 16 / 16 64 / 64
memory (req/lim) 192 Gi / 192 Gi 192 Gi / 192 Gi 384 Gi / 384 Gi
nvidia.com/gpu 1 1 2
PVC count 5 5 10

Prod sc-task-queue was bumped to 2 GPUs in commit 9e4a06d; biotools jupyter was scaled 6→4 GPU in daa00d0 to free useast13-prod capacity for that bump.

sc-task-queue Resources

Env requests limits GPU
base 4 cpu / 32 Gi 8 cpu / 64 Gi 1
dev / qa overlay 4 cpu / 16 Gi 8 cpu / 96 Gi inherited (1)
prod overlay 4 cpu / 16 Gi 8 cpu / 192 Gi 2

Node placement: base manifest pins nodeSelector: compute.coreweave.com/node-pool=gpu-rtx6000p. The dev overlay patches this to barad-dur (the only pool on useast13-dev).

Networking

MongoDB

Single-pod StatefulSet running mongo:7.0.5 --replSet rs0 --bind_ip_all. The readiness probe self-initiates the replica set on first boot using the pod's stable DNS name. Both a headless service (mongodb — required for replica-set addressing and Meteor oplog tailing) and a ClusterIP service (mongodb-svc) exist.

The base manifests hardcode iscanguide.svc.cluster.local in two string values that Kustomize's namespace transformer cannot rewrite: MONGO_OPLOG_URL and the rs.initiate host in the readiness probe. The dev and qa overlays patch these to their own namespace; prod is the namespace iscanguide, so no patch is needed.

A switch to MongoDB Atlas is planned — the StatefulSet's leading comment calls it interim.

Secrets

app-secrets in each namespace carries:

secrets.yaml is gitignored. Each overlay carries a secrets.yaml.example. The qa/prod .example files note that they should be pulled from AWS Secrets Manager.

UsersServiceUrl

The address iSCanGuide uses to reach the prod-services users service (:10007), set in meteor-settings as a raw host IP per environment:

Context / namespace UsersServiceUrl
useast13-prod / iscanguide 34.218.78.129:10007
useast13-prod / iscanguide-qa 35.87.104.177:10007
useast13-dev / iscanguide-dev 34.220.189.245:10007
useast13-dev / iscanguide-qa 34.220.189.245:10007

users has no application auth; access is controlled by the prod-services host security group sg-dd1155ba, which admits :10007 from 166.19.88.0/24 (Coreweave — iSCanGuide's egress). The SG rule applies to whichever hosts run users; the UsersServiceUrl IP names one host, so it is updated when that host is replaced. A stable address is users behind the front ALB :8443 listener (SG-gated, admitting 166.19.88.0/24), with UsersServiceUrl pointed at its DNS name.

Deploy scripts

projects/iscanGuide/scripts/:

k8s/overlays/<env>/setup.sh is the first-time bootstrap for an environment (namespace, ECR auth, secrets, kustomize build, LB IP wait, ROOT_URL patch, cert wiring). deploy.sh / deploy-prod.sh are the routine image-update path.

k8s/cluster-bootstrap.sh is the one-shot cluster prep (ESO, cert-manager ClusterIssuer, priority classes, source-range allowlist).

Repo layout assumption

The deploy scripts assume app source repos sit alongside this repo:

~/Projects/
  advaita-deploy/      ← configs and scripts (this repo)
  iscanGuide/
    sc-website/        ← Meteor/React app
    sc-task-queue/     ← Node dispatcher
    sc-website-scripts/ ← R/Seurat (baked into sc-worker image)

The deploy script also accepts SC_WEBSITE_DIR override.

Outstanding work

What to verify against the live cluster (once read-only access lands)

Sibling products under the same deploy tree

For later — these share the cluster but are distinct products:

Deploy structure — platform + per-service bundles

Greenlit 2026-07-15 (branch feature/iscanguide-kustomize-split): the single shared kustomize overlay is split into four independent, top-level kustomize bundles, each with its own dev/qa/prod overlays and — for the services — its own Vivarta pipeline.

Bundle Owns
iscanguide-platform the namespace + shared infrastructure: rbac, the shared PVCs (data-pvc / params-pvc / logs-pvc), network-policy, mongodb (StatefulSet + Service), resource-quota, TLS certificate/issuer
sc-website sc-website Deployment + Service + Ingress
sc-task-queue sc-task-queue (worker) Deployment + Service
sc-server-zoom server-zoom Deployment + Service

The iscanguide-platform component is the namespace's shared foundation. The three service bundles reference its resources by name (data-pvc, ecr-pull-secret, the mongodb Service), so it is applied first on a fresh namespace. It carries no image, so it is applied with kubectl apply -k rather than a build pipeline; the service bundles each get a pipeline that applies only that service — so a deploy no longer re-asserts the other services' image tags the way applying the shared overlay did.

Two shared patches split into per-bundle fragments in the process: patch-mongo-dns (the mongodb StatefulSet half to platform, the sc-website MONGO_OPLOG_URL half to sc-website) and patch-priority-class (one fragment per service + the mongodb fragment in platform). Full mapping and adoption steps: projects/ISCANGUIDE-KUSTOMIZE-SPLIT.md.

Build / deploy / verify (runbook)

Verified against the live useast13-dev cluster on 2026-05-27. The build runs on Jenkins; the deploy runs from a bastion via the kustomize bash scripts. Both are slated to move to Vivarta (next section).

Build (Jenkins)

Images are built by the Jenkins job sc-task-queue on the jenkins.cw.advaitabio.com controller (CoreWeave cluster). The Jenkinsfile (sc-task-queue repo, develop branch) checks out two repos side by side — sc-task-queue (primary) and sc-website-scripts (sibling) — then builds and pushes two images to ECR us-west-2 (789905347053):

Each push gets a precise tag <version>-<sha>[-<sha>] plus a floating <version> tag, e.g. sc-worker:1.9.0-SNAPSHOT-aabb18a-047d0b4 and sc-worker:1.9.0-SNAPSHOT. A green build ends with ✓ … pushed / ✓ floating tag lines.

Run it: open the job and Build Now, or vivarta-jenkins/scripts/jenkins build sc-task-queue --watch. The docker-buildx step is long (~25 min incl. the worker layers) and its mid-step output does not stream to the Jenkins console until the step finishes (pipeline log buffering) — for live output use jenkins-logs --agent <pod> against the agent pod (kubectl), not the HTTP console.

Deploy (bastion + kustomize)

Deploys run from a bastion EC2 host, not a laptop:

mssh ubuntu@i-08b2d93fde189764b --region us-west-2
cd /mnt/build/advaita-deploy/projects/iscanGuide

scripts/deploy.sh is an interactive ECR SNAPSHOT-tag picker: it lists SNAPSHOT tags from ECR, updates the kustomize image tags + buildNumber in secrets.yaml, applies the overlay, and rollout-restarts the deployments.

bash scripts/deploy.sh --env=dev                # both images to dev
bash scripts/deploy.sh --env=qa                 # both images to qa
bash scripts/deploy.sh --env=dev --worker-only  # sc-worker only
bash scripts/deploy.sh --env=dev --website-only # sc-website only
bash scripts/deploy.sh --env=dev --deploy-only  # re-apply, no image change
bash scripts/deploy.sh --env=dev --dryrun       # preview

Per-env wiring (scripts/env/*.env):

env namespace kube context URL
dev iscanguide-dev useast13-dev https://iscanguide-dev.cw.advaitabio.com
qa iscanguide-qa useast13-dev https://iscanguide-bd.advaitabio.com
prod iscanguide useast13-prod https://iscanguide.advaitabio.com

dev and qa share the useast13-dev cluster (separate namespaces); prod is useast13-prod. The script aborts if kubectl current-context doesn't match the env's KUBE_CONTEXT. Prereqs: kubectl, kustomize, awscli, jq, and secrets.yaml populated from AWS Secrets Manager.

Verify

CTX=useast13-dev; NS=iscanguide-dev
kubectl --context $CTX -n $NS rollout status deployment/sc-task-queue
kubectl --context $CTX -n $NS get pods -l app=sc-task-queue -o wide
kubectl --context $CTX -n $NS get deploy sc-task-queue \
  -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'   # confirm the tag landed

The rollout's slow step is the image pull — the 5.5 GB sc-worker layer pulls in ~2m43s, during which rollout status reports 0 of 1 updated replicas are available. That's the pull, not a stall; then the new pod goes 1/1 Running and the old ReplicaSet scales to 0. Finish by hitting the env APP_URL.

Live verification (useast13-dev / iscanguide-dev, 2026-05-27)

Running images: iscanguide/sc-worker:1.9.0-SNAPSHOT-aabb18a-047d0b4, iscanguide/sc-website:1.9.0-SNAPSHOT-21b9250 (2 replicas), app-monitor:0.1.6. Workloads, services, ingress (sc-website nginx → iscanguide-dev.cw.advaitabio.com, 166.19.90.28), and PVCs (data 500Gi / logs 8Gi / params 3Gi RWX + mongo-data 20Gi RWO, all shared-vast) all match the config-derived baseline above — so the dev environment is now live-verified, not config-only. Notable: dev-quota shows 1/1 GPU in use (the worker is GPU-scheduled), and a ecr-token-sync ServiceAccount keeps the in-cluster ECR pull creds fresh.

Production deploy — play-by-play (deploy-prod.sh)

The prod release path is scripts/deploy-prod.sh (ECR-first: it promotes a QA-verified SNAPSHOT image to a prod tag and applies — it never builds). Derived from the script + lib.sh + env/prod.env on develop (2026-05-29); config-derived, not yet live-verified against useast13-prod.

Before you start

Run it (pick exactly one image):

bash scripts/deploy-prod.sh --website-only        # interactive SNAPSHOT pick
bash scripts/deploy-prod.sh --worker-only
bash scripts/deploy-prod.sh --website-only --snapshot-tag 1.9.0-SNAPSHOT-aabb18a
bash scripts/deploy-prod.sh --website-only --dryrun     # no ECR retag, no K8s mutation

What the script does, in order:

Re-apply without an image change: bash scripts/deploy-prod.sh --website-only --deploy-only reads the current tag from kustomization.yaml, runs preflight + the production gate, then applies + rolls out + annotates + commits — no ECR retag, no image/git changes.

Verify after:

CTX=useast13-prod; NS=iscanguide
kubectl --context $CTX -n $NS rollout status deployment/sc-website        # or sc-task-queue
kubectl --context $CTX -n $NS get pods -o wide
kubectl --context $CTX -n $NS get deploy sc-website \
  -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'           # confirm PROD_TAG landed

Then hit https://iscanguide.advaitabio.com. Remember prod and qa share useast13-prod (namespace + quota + NetworkPolicy + PriorityClass isolation), so a wrong-context or wrong-namespace mistake lands in the live cluster — the preflight context gate is the guard.

Migration to Vivarta (planned)

The Jenkinsfile build and the bash+kustomize deploy both move onto the Vivarta pipeline ars/aem now use. Shape:

Open question before porting: where the kustomize image-tag bump + secrets.yaml (from AWS Secrets Manager) live in the Vivarta model — likely a new boundary, since the ECS path used SSM + CFN params rather than kustomize edits and a Secrets-Manager-sourced secrets.yaml.

What's running per environment (prep / read-only)

Run before a deploy to capture the current image baseline. All read-only, so they don't trip the deploy preflight's context gate. Contexts + namespaces (from scripts/env/*.env, the source of truth):

env --context -n namespace
dev useast13-dev iscanguide-dev
qa useast13-dev iscanguide-qa
prod useast13-prod iscanguide

Quick glance — image tag per deployment, all three at once:

for spec in "useast13-dev iscanguide-dev" "useast13-dev iscanguide-qa" "useast13-prod iscanguide"; do
  set -- $spec; CTX=$1; NS=$2
  echo "── $NS ($CTX) ──"
  kubectl --context "$CTX" -n "$NS" get deploy sc-website sc-task-queue \
    -o custom-columns='DEPLOY:.metadata.name,IMAGE:.spec.template.spec.containers[0].image' --no-headers 2>/dev/null
done

Exact bytes — running pods' tag vs resolved digest (same dump the deploy script prints; mutable tags can drift from the digest, so this is the real "what's live"):

CTX=useast13-prod; NS=iscanguide        # swap per the table above
kubectl --context "$CTX" -n "$NS" get pods -l 'app in (sc-website,sc-task-queue)' -o json \
  | jq -r '.items[] | select(.status.phase=="Running") | .metadata.name as $p
           | (.status.containerStatuses // [])[]
           | "  \($p)\n    image:  \(.image)\n    digest: \(.imageID)"'

What the deploy overlay would set (compare against the above to see if a redeploy would change anything):

grep -A2 'images:' k8s/overlays/prod/kustomization.yaml      # newName / newTag per image