Wanderland

advaita-nextflow-router-data-model

How one request becomes a queued Kubernetes workload — written as a ledger: what each stage consumes, what it produces, and which channel carries the result downstream.

Fills the seam the parent flags — advaita-nextflow-architecture §10 ends "Not present yet: … any queue / scheduling state — a run is fire-and-forget, there is no queue." This node is that queue and the router in front of it. Design in case task-55de866f; the running service is advaita-service-wisp; idiom in wanderland-dev-ai-primer.

The one idea

The thing that enters the queue is a single shape — a batch/v1 Job, and the task Pods it spawns. Kueue reads a few fields off a workload: the kueue.x-k8s.io/queue-name label, the container resources.requests, an optional priorityClassName. Everything else is payload.

So job types never differ in shape — only in values. A record starts small and semantic at the client and is progressively hydrated until it is a complete Job. The key-set stays fixed, and each field names the source that fills it.

Provenance legend

Tag Source When
[C] Client — the calling app sends it request time
[I] Identity — the Wanderland from_addr / the minted run_id request time (platform)
[F] Foxfire — recorded history, fetched live: the envelope, resume lineage classify time
[P] Profiler — computed by wisp over [F] numbers within [Y] bounds classify time
[Y] YAML — the template library (values + placeholders) boot-loaded, resolved at dispatch
[K] Kueue — injected by the scheduler admission time

[F] and [P] are distinct on purpose. Foxfire supplies numbers ("231MB p95 over 9 runs"); wisp computes decisions over them ("tier sm, queue workers-deferred-cpu, spot-eligible"). The numbers are recorded truth; the decision is wisp's — and the decision is itself recorded (the plan, Stage G), so it can be checked against what actually happened.

The transports

Six channels carry data down the stack. Every stage writes into one or more of them; nothing else crosses a boundary. An integration adds fields to these channels, not new channels.

Channel Written by Read by Carries
params-file (params.json) hydrate → classify Nextflow as params the header/data/tenancy document — knobs, inputs, tenancy
argv (ERB-hydrated) classify driver process run options, workflow + config paths, -params-file, -resume <session-id>
driver env classify driver + the workflow's nextflow.config (System.getenv) foxfire endpoint + key id, RUN_NAME, executor targets (NF_*), NXF_CACHE_DIR, callback URLs
Job metadata classify Kueue (driver admission), reaper, debug UIs queue-name: drivers, run-id, app labels
overlay ConfigMap classify, from profile.per_process Nextflow config merge → every task pod per-process cpus / memory / accelerator + resourceLabels (queue-name, run-id, process)
volumes classify, from template + config driver filesystem work PVC, cache PVC, data PVCs, the overlay itself, the projected identity token

The params-file and the overlay share one transport: both are keys in the per-run ConfigMap (params.json, overlay.config) mounted at /etc/wisp, so data, tenancy, and the knobs reach the workflow without a new volume or a per-field argv hole. params-file supersedes the older idiom of hydrating every knob into a hand-written --flag <%= params.x %> argv pair — the ERB run_opts still work for templates that use them, and a workflow reading params.data.* needs only the file.

Two of these have jobs worth naming:

The overlay is the second-hop control surface. wisp never sees the task pods — Nextflow's k8s executor creates them one hop down. Its decision reaches them because the driver runs nextflow -c base.config -c /etc/wisp/overlay.config and Nextflow merges the per-run fragment over the workflow's baked config. Each process gets its wisp-decided resources and its queue-name label through a config merge; no workflow code changes. Because the label is per-process, two processes in one run can route to different worker queues. The overlay is a full Nextflow config file — today it carries only process {} directives, but the merge semantics carry any scope, which is where per-run app config (an aws {} endpoint block, a future app scope) lands without inventing a channel.

The projected identity token is the credential channel. A template may declare driver.identity_token: { audience: … }; classify then mounts an audience-scoped, kubelet-rotated ServiceAccount token at /var/run/secrets/app-identity/token. Nothing sensitive crosses argv, env, or .command.sh — the credential is a file the kubelet refreshes before expiry, so long deferred/batch runs stay authenticated with no static secret. The audience scoping means the token authenticates to exactly one consumer (a TokenReview-validating app such as DataHub) and replays nowhere else.

A boundary note on params: anything the client passes there surfaces in the driver Job's argv via ERB — visible to anyone who can read the Job spec. Parameters are for names and knobs. Credentials go through the token channel; secrets go through Secret refs.

The stages

client ─▶ submit ─▶ envelope ─▶ profile ─▶ classify ─▶ Kueue ─▶ driver ─▶ task pods
  [C]      [C+I]      [F]        [+P]     [+Y hydrate]   [K]         │
                                              │                      ▼
                                              └─ plan ─────▶ foxfire ◀─ placement (reaper)
                                                                     ◀─ trace (nf-foxfire)

Stage A — what the client sends (POST /submit)

consumes:  the caller's intent
produces:  the request record [C]
transport: POST body

The generic low-level route. The client sends the what, not the how.

# request body to POST /submit
app_id:    img-website        # [C] who is calling (also on the crossing identity)
job_type:  img.recontrast_run # [C] the KIND of work — the key the template when: matches
immediacy: deferred           # [C] DECLARED: immediate | deferred | batch (default deferred)
params:                       # [C] the knobs — workflow settings; flattened into params.header
  study:   ST000405
  case:    PA
  control: No PA
data:                         # [C] the input map — named keys → URIs (hub:// s3:// http:// ftp://)
  sample:  hub://acme/proteomics/x.mzML
  ref_db:  s3://public-refs/uniprot/2026.fasta
tenancy:                      # [C] the tenancy triple — who the run belongs to
  org_id:   acme
  team_id:  proteomics        #     team_id optional (a file's pinned team)
  owner_id: u-31f2
size_hint: null               # [C] optional caller override (the bump-up lever)
resume_of: null               # [C] run_id of a prior run to CONTINUE — Stage E resolves
                              #     its Nextflow session from foxfire lineage
callback_url: null            # [C] endpoint(s) that also receive the run's signed trace
                              #     events (a URL or comma-list) — event-driven client
                              #     updates instead of polling

Three of these carry the workload rather than route it: params (knobs), data (inputs), and tenancy (org/team/owner). They never touch the routing decision — profile sizes from the envelope, classify matches on job_type — but they are validated and shaped, by the hydrate boundary, into the one document the workflow reads (Stage B). data values are scheme-dispatched by Nextflow: hub:// through the platform's path factory, s3:///http/ftp through stock providers. tenancy binds the run to an org/team/owner; the org selects the driver's ServiceAccount at launch, and the triple is recorded for the storage plane's per-file authorization (advaita-datahub-storage-data-model).

immediacy is when it needs to run, declared by the caller (not measured):

It biases placement — which worker queue, and the priority/preemption between them — but it does not size the work; that is the envelope's job. A job declared immediate that history shows runs 45 minutes can still be sized large and routed accordingly; the immediacy only says how soon it must start.

resume_of names a specific prior run. Resume is always explicit — the caller says continue that run, never "continue whatever ran last" (on a shared work volume, "last" is a race between every driver on the cluster).

callback_url registers the caller as a consumer of the run's trace events (Stage G): the driver fans each event to every listed endpoint.

The callback projection. Each registered endpoint chooses the shape it receives: raw (the verbatim EC-P256-signed envelope the foxfire receiver ingests) or flattened (a normalized, unsigned record — the event fields hoisted to the top level, exit_status surfaced, the nested payload and signature dropped), written as a url|projection suffix on the list and defaulting to flattened. A client that wants pushed progress without handling the signed envelope registers flattened; one that verifies at the edge registers raw. Stage G is the wire contract; the flattened field mapping lives in foxfire's docs/event-envelope.md.

Stage B — submit seeds the context, hydrate shapes the params document

consumes:  the request record [C]
produces:  run_id [I]; the params document (header/data/tenancy) [C-shaped]
transport: request context (!Context); the params document → params.json in the per-run ConfigMap

submit validates the body, mints the run_id, and writes the request into the context.

# result of the submit boundary (read downstream as for_boundary.submit.*)
job_type:    img.recontrast_run          # [C]
immediacy:   deferred                    # [C]
params:      { … }                       # [C]
app_identity: "app:img-website"          # [I] the crossing from_addr (who called)
run_id:      img-20260707210357-a1b2c3   # [I] <lane>-<UTC stamp>-<entropy>

The run_id is a triple identity: the Job name, the Nextflow -name, and the run-id label — DNS-1123 safe, minted once. Every later record joins on it: the plan, the placement, the trace, the resume lineage, and (for app auth) the pod name a TokenReview surfaces to a caller verifying "this is run X's driver".

Then hydrate shapes the request into the one document the workflow receives, params.json, and structurally validates it — with no template in play. It knows nothing about job types, sizing, or queues; it checks that data is a map of URI strings and that a non-empty tenancy carries its fence (org_id, owner_id). The run fields sit at the top; params, data, and tenancy are sibling maps:

# params.json — mounted at /etc/wisp/params.json, read by Nextflow as `params`
run_id:    img-20260707210357-a1b2c3     # the run envelope, top-level scalars
job_type:  img.recontrast_run
immediacy: deferred
params:                                  # [C] the knobs — params.params.study in the workflow
  study:     ST000405
  threshold: "0.05"
data:                                    # [C] inputs — params.data.sample → file()
  sample:  hub://acme/proteomics/x.mzML
  ref_db:  s3://public-refs/uniprot/2026.fasta
tenancy:                                 # [C] tenancy — params.tenancy.org_id
  org_id:   acme
  team_id:  proteomics
  owner_id: u-31f2

Because this shaping needs no template, it is testable on its own: the /validate route is submit → hydrate and nothing else — "is this request well-formed, and what will the workflow receive?" — answered before foxfire, sizing, or a when: match. Template-schema validation (are these the params this job_type declares?) is a later, separate concern.

Stage C — the envelope, then profile

consumes:  envelope [F]              (per-process p95 stats; 404 = cold start)
           template resource class [Y] (family, min/max clamps, cold defaults, spot posture)
           immediacy [C]
produces:  per_process [P]; run headline { size, compute, queue_name } [P]
transport: !Context → classify.args

foxfire_envelope fetches the raw per-process statistics foxfire holds for this job_type; profile matches them to a size. Foxfire never says "large" — it says "231MB p95"; wisp decides the size from the numbers. The envelope is a plain roll-up of the ledger (real useast13-dev data, 9 runs):

# foxfire envelope — img.recontrast_run, raw stats per process   [F]
processes:
  RECONTRAST: { cpus: 1, realtime_ms: { p95: 125 },   peak_rss_bytes: { p95: 13700000 } }
  ORA:        { cpus: 1, realtime_ms: { p95: 2612 },  peak_rss_bytes: { p95: 231500000 } }
  RAMP:       { cpus: 1, realtime_ms: { p95: 15448 }, peak_rss_bytes: { p95: 147700000 } }
  PERTURB:    { cpus: 1, realtime_ms: { p95: 9629 },  peak_rss_bytes: { p95: 114400000 } }

The matcher (Wisp::Sizing) picks, per process, the smallest tier whose allocation covers p95 peak_rss × headroom and the observed cpus — on two axes, family (cpu | gpu) × tier — then the template's resource class clamps the result to its declared min/max. Profile sizes the union of the envelope's processes and the template's declared task_processes, so a process with no history is still sized:

The per-process record profile emits is the full decision plus the evidence it was made from:

# profile output — per_process (one entry per process)   [P]
per_process:
  ORA:
    tier: sm                                  # matched tier (post-clamp)
    family: cpu                               # cpu | gpu — gpu is DECLARED in the class
    requests: { cpu: "1", memory: 512Mi }     # k8s requests — what Kueue admits on
    nf:       { cpus: 1, memory: '512.MB' }   # Nextflow directives for the overlay
    queue: workers-deferred-cpu               # immediacy [C] × family
    spot_eligible: false                      # the spot DECISION (see below)
    flavor: on-demand                         # spot | on-demand, from that decision
    cold: false                               # sized from history, not class defaults
    samples: 9                                # ── the evidence ──
    rss_p95_mb: 231
    rt_p95_ms: 2612
size: sm                                      # run headline: top tier across processes
compute: cpu                                  # gpu if any process pulled one
queue_name: workers-deferred-cpu              # the headline queue

Spot is decided here, per process, as posture-then-evidence:

Enforcement is a downstream concern — flavors and node constraints where spot pools exist — but the decision is a [P] output, carried on the plan so drift is visible.

Stage D — the template library ([Y]), a when:-guarded step

consumes:  job_type [C] (the when: match)
produces:  driver values [Y] — image, paths, run_opts holes, mounts,
           resource class, identity_token
transport: boot-loaded config (!Fixture), matched at dispatch

A template is a library entry: a when: predicate (shape-matched by Wanderland::ShapeMatcher against the request) plus the driver values and the resource class. Adding a job type is a new entry — no code, no rebuild.

# config/templates.yml (one entry)
templates:
  - id: img.recontrast_run
    when:                                  # [Y] ShapeMatcher against the request facts
      job_type: img.recontrast_run
    driver:
      image: "…/ortho-driver:latest"       # [Y]
      command: [nextflow]                  # [Y]
      config_path:   /workflows/ortho/nextflow.config   # [Y] base config classify -c's
      workflow_path: /workflows/ortho/main.nf           # [Y]
      run_opts:                            # [Y] argv after `run <flow>`; ERB holes → [C]
        - -profile
        - kubernetes
        - -name
        - "<%= run_id %>"
        - --study
        - "<%= params.study %>"
        - --case
        - "<%= params.case %>"
        - --control
        - "<%= params.control %>"
      mounts:                              # [Y] data volumes on the driver
        - { pvc: ortho-dbs,  path: /opt/img/source_cache, read_only: true }
        - { pvc: ortho-data, path: /opt/img/img-processed-20260610 }
        - { pvc: nextflow-work, path: /work }
      # identity_token:                    # [Y] OPT-IN credential channel — app templates
      #   audience: datahub                #     only; mounts a projected SA token
    resources:                             # [Y] the resource class
      default:                             #     applies to the flow + every process
        cpu:    { default: 1,    min: 1,   max: 4 }
        memory: { default: 1024, min: 512, max: 16384 }
        gpu:    { default: 0 }             #     gpu family is DECLARED here, never inferred
      processes:
        ORA:                               #     named override — p95 RSS well above the rest
          spot: unsafe                     #     long + uncheckpointed: pinned on-demand
          memory: { default: 2048, min: 1024, max: 32768 }
    task_processes: [RECONTRAST, ORA, RAMP, PERTURB]   # sized even with no history

classify matches the first template whose when: fits, ERB-hydrates run_opts against { run_id, params }, and assembles the command: nextflow -c <config_path> -c /etc/wisp/overlay.config run <workflow_path> <run_opts…>. (Implementation note: wanderland ERB-renders config + !Fixture files at load, so the holes are written <%%= … %> in the file — the load pass emits a literal <%= … %> that classify evaluates.)

Stage E — classify: the hydrated Job

consumes:  run_id [I], per_process [P], template [Y], params [C],
           resume_of [C] → session id [F]
produces:  overlay ConfigMap + driver Job (applied to the cluster);
           the plan → foxfire (Stage G)
transport: kubectl apply; POST <foxfire>/plan

Resume resolution. A submit carrying resume_of: <run_id> makes classify resolve that run's Nextflow session from foxfire's lineage record and append -resume <session-id> to argv. Unresolvable lineage is a 422, never a silent fresh run — a resume that quietly re-computes everything is a lie about what happened. Session cache DBs live on the always-shared cache PVC (NXF_CACHE_DIR=/nxf-cache), so lineage resolves the same way whatever the launch-dir topology.

The overlay, rendered from profile.per_process into a per-run ConfigMap (<run_id>-overlay):

// /etc/wisp/overlay.config — rendered by wisp from profile.per_process
process {
    withName: 'ORA' {
        cpus = 1
        memory = '512.MB'
        resourceLabels = ['kueue.x-k8s.io/queue-name': 'workers-deferred-cpu',
                          'run-id': 'img-20260707210357-a1b2c3',
                          'process': 'ORA']
    }
    // … one withName block per process …
}

Three labels per task pod: queue-name gates it through Kueue; run-id + process are the join keys the reaper reads to tie observed placement back to this run's plan.

The Job (built as hashes → YAML by Wisp::JobManifest): a ConfigMap holding the overlay, plus the driver Job that mounts it.

apiVersion: batch/v1
kind: Job
metadata:
  name:      img-20260707210357-a1b2c3
  namespace: vivarta-drivers                        # [Y/env] driver ns → the drivers queue
  labels:
    app: wisp-driver
    run-id: img-20260707210357-a1b2c3               # [I]
    kueue.x-k8s.io/queue-name: drivers              # gated in its OWN pool — caps workflows
spec:
  suspend: true                                     # Kueue admits it off the queue
  backoffLimit: 0                                   # a failed driver is a failed run — no retry storm
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: nextflow-runner           # [Y/env]
      containers:
        - name: driver
          image:   "…/ortho-driver:latest"          # [Y]
          command: [nextflow]
          args: [ -c, …/nextflow.config, -c, /etc/wisp/overlay.config,
                  run, …/main.nf, --study, ST000405, … ]        # [Y]+[C] (+ -resume [F])
          resources: { requests: { cpu: 500m, memory: 1Gi } }   # driver small, fixed
          env:
            - FOXFIRE_URL: …/events/signed          # trace sink (signed events)
            - FOXFIRE_KEY_ID: nf-foxfire
            - FOXFIRE_SIGNING_KEY: (secretKeyRef)   # EC key nf-foxfire signs with
            - RUN_NAME: img-20260707210357-a1b2c3   # [I]
            - NF_NAMESPACE: vivarta-workers         # k8s executor: where task pods land
            - NF_SERVICE_ACCOUNT: nextflow-runner
            - NF_WORK_PVC: nextflow-work            # shared work volume claim
            - NXF_CACHE_DIR: /nxf-cache             # session cache home → resume lineage
            - FOXFIRE_CALLBACK_URLS: …              # [C] optional event fan-out
          volumeMounts:
            - { name: wisp-overlay, mountPath: /etc/wisp }
            - { name: nxf-cache,    mountPath: /nxf-cache }
            # - { name: app-identity-token, mountPath: /var/run/secrets/app-identity }
      volumes:
        - { name: wisp-overlay, configMap: { name: img-20260707210357-a1b2c3-overlay } }
        - { name: nxf-cache, persistentVolumeClaim: { claimName: nextflow-cache } }
        - { name: vol0, persistentVolumeClaim: { claimName: nextflow-work } }
        # … template data PVCs; the projected identity token when the template declares it

The task pods the driver spawns land in vivarta-workers, each stamped by the overlay with queue-name: workers-<immediacy>-<compute> + its per-process requests — gated against the separate workers pool.

The plan. After the apply, classify POSTs the full decision to foxfire's /plan ingest: run_id, job_type, immediacy, template id, headline size/compute /queue, the driver's image + namespace + queue, and the complete per_process map — decisions and the statistics they were sized from. Best-effort (a launch never fails on lost telemetry); Stage G says why it matters.

Stage F — into the queue

consumes:  the driver Job, then each task pod — each carrying a queue-name label
produces:  admission (or queueing) against the flavor quotas; node placement [K]
transport: Kueue ClusterQueues / LocalQueues

Both the driver Job and every child pod carry a queue-name; Kueue admits each against its ClusterQueue's quota. The deferred/batch worker queues are spot-first and spill to on-demand via flavorFungibility: TryNextFlavor; the immediate queue is on-demand only. At admission Kueue injects the placement:

# what Kueue adds at admission (a spot-eligible worker pod)
spec:
  nodeSelector: { node-class: spot }                  # [K]
  tolerations:  [ { key: spot, operator: Exists } ]   # [K]

(On OrbStack both flavors match the single node — the flavor logic is exercised without real pools; the nodeSelector injection places real pools where node-class labels exist.)

Two pools, two throttles, in separate cohorts so they never borrow across:

Worst case is serialization (drivers waiting while their tasks trickle through the worker pool), which is backpressure, not deadlock — tasks always make progress because their quota is disjoint from the drivers'. Keeping the two cohorts separate matters: one cohort would let Kueue lend the drivers the workers' idle quota, which re-creates the shared-pool wedge.

Stage G — the ledgers (how the loop closes)

consumes:  the run's lifecycle
produces:  three ledgers in foxfire, joined on run_id (+ process):
             plan       [P→F]  what wisp decided, and from what evidence
             placement  [F]    where each pod actually landed (reaper)
             trace      [F]    what each task actually used (nf-foxfire, signed)
transport: POST /plan (wisp) · pod watch (reaper) · EC-P256-signed events
           (nf-foxfire) + callback fan-out (FOXFIRE_CALLBACK_URLS)

The signed event stream and these ledgers are the authoritative record. Everything downstream is a derived view of them: the plan-vs-placement drift view, the UI's fold (the launcher reads the fold and only the fold — never Kueue, never the driver), the envelope that sizes the next run of this job_type, and every callback consumer. The fan-out is shaped per endpoint by a projection: FOXFIRE_CALLBACK_URLS registers each endpoint as raw — the verbatim EC-P256-signed envelope, which the foxfire receiver always takes and verifies — or flattened, a normalized, unsigned record with the event fields hoisted to the top level (run_id / session_id / utc_time, and on run:completed a top-level exit_status), selected with a url|projection suffix and defaulting to flattened. A raw subscriber holds a byte-exact, verifiable copy of what foxfire folds; a flattened subscriber holds a convenience view and reconciles integrity against foxfire's read surface (GET /runs/:run_id + /verify). Neither carries a per-run seq — foxfire assigns that on ingest — so a subscriber correlates on run_id and treats the fan-out as best-effort delivery of a stream whose authoritative copy always exists at foxfire. Views that disagree reconcile by going back to the record, not by arbitrating with each other (lebowski-corollary). The loop closes when the next classification reads the roll-up of this run's trace.

Worked example 2 — a real run, end to end (echo.fibfizz)

echo-20260717144738-d94b94, submitted to the live wisp on useast13-dev on 2026-07-17 as an immediate run. Every value below is fetched, not authored: the overlay and Job are the applied objects read off the cluster before their TTL; the plan, placement, and trace are read back from foxfire's ledgers. Its sibling echo-20260716192138-12d736 — same job type, submitted deferred the day before — appears where the immediacy axis makes the two runs differ. The Job shape is identical to the recontrast walk-through above; only the values change.

A — the request

The echo launcher asks for fibfizz of x: 25, with a caller waiting on it.

consumes:  the launcher form
produces:  the request record [C]
transport: POST /submit body

job_type:  echo.fibfizz   # [C]
immediacy: immediate      # [C]  (the sibling declared deferred)
params: { x: "25" }       # [C]

B — submit

The run_id is minted at 14:47:38 UTC; every later record joins on it.

consumes:  the request record [C]
produces:  run_id [I]; the validated record in the context
transport: request context (!Context)

run_id:    echo-20260717144738-d94b94   # [I] echo-<UTC stamp>-<entropy>
job_type:  echo.fibfizz                 # [C]
immediacy: immediate                    # [C]
params:    { x: "25" }                  # [C]

C — the envelope, then profile

The live roll-up holds 15 samples per process; both match sm/cpu. Spot rule 1 fires — immediate work is never spot — so spot_eligible: false however short the runtimes. (The deferred sibling, same numbers, derived spot_eligible: true: rt p95 ≈2.5–3.5s, far under the 300s threshold. The immediacy axis, not the envelope, is what changed.)

consumes:  envelope [F] (live rollup, 15 samples/process), resource class [Y],
           immediacy [C]
produces:  per_process [P]; headline { size: sm, compute: cpu,
           queue_name: workers-immediate-cpu }
transport: !Context → classify.args

# the envelope [F] — foxfire live rollup for echo.fibfizz
processes:
  FIBONACCI: { samples: 15, cpus: 1, gpu: 0, fanout_per_run: 1.0,
               realtime_ms: { mean: 3450, p95: 3458 },
               peak_rss_bytes: { mean: 44951689, p95: 45113344 } }
  FIZZBUZZ:  { samples: 15, cpus: 1, gpu: 0, fanout_per_run: 1.0,
               realtime_ms: { mean: 2452, p95: 2475 },
               peak_rss_bytes: { mean: 44700467, p95: 46157824 } }

# profile output [P] — as recorded on this run's plan
per_process:
  FIBONACCI: { tier: sm, family: cpu, requests: { cpu: "1", memory: 512Mi },
               nf: { cpus: 1, memory: '512.MB' }, queue: workers-immediate-cpu,
               spot_eligible: false, flavor: on-demand, cold: false,
               samples: 15, rss_p95_mb: 45, rt_p95_ms: 3458 }
  FIZZBUZZ:  { tier: sm, family: cpu, requests: { cpu: "1", memory: 512Mi },
               nf: { cpus: 1, memory: '512.MB' }, queue: workers-immediate-cpu,
               spot_eligible: false, flavor: on-demand, cold: false,
               samples: 15, rss_p95_mb: 46, rt_p95_ms: 2475 }
size: sm
compute: cpu
queue_name: workers-immediate-cpu

D — the template

when: { job_type: echo.fibfizz } matches the nf-driver entry; ERB fills the two holes — the run name and --x.

consumes:  job_type [C] (the when: match); { run_id, params } for the holes
produces:  driver values [Y]; the assembled argv
transport: boot-loaded config, resolved at dispatch

# argv as applied on the cluster (verbatim from the Job)
image: …/advaita/nf-driver:latest              # [Y]
argv:
  - -c
  - /workflows/fibfizz/nextflow.config          # [Y] the baked config
  - -c
  - /etc/wisp/overlay.config                    # the wisp overlay (merged second)
  - run
  - /workflows/fibfizz/main.nf                  # [Y]
  - -profile
  - kubernetes
  - -name
  - echo-20260717144738-d94b94                  # [I] via ERB
  - --x
  - "25"                                        # [C] via ERB

E — classify

The overlay ConfigMap and driver Job are applied; the plan is POSTed to foxfire in the same call. Both blocks below are the applied objects, fetched from the cluster while the run lived.

consumes:  run_id [I], per_process [P], template [Y], params [C]
produces:  echo-20260717144738-d94b94-overlay ConfigMap + driver Job (applied);
           the plan → foxfire
transport: kubectl apply; POST /plan

// /etc/wisp/overlay.config — verbatim from the applied ConfigMap
process {
    withName: 'FIBONACCI' {
        cpus = 1
        memory = '512.MB'
        resourceLabels = ['kueue.x-k8s.io/queue-name': 'workers-immediate-cpu', 'run-id': 'echo-20260717144738-d94b94', 'process': 'FIBONACCI']
    }
    withName: 'FIZZBUZZ' {
        cpus = 1
        memory = '512.MB'
        resourceLabels = ['kueue.x-k8s.io/queue-name': 'workers-immediate-cpu', 'run-id': 'echo-20260717144738-d94b94', 'process': 'FIZZBUZZ']
    }
}

# the driver Job — verbatim (env + volumes in full; boilerplate elided)
metadata:
  name: echo-20260717144738-d94b94
  namespace: vivarta-drivers
  labels: { app: wisp-driver, run-id: echo-20260717144738-d94b94,
            kueue.x-k8s.io/queue-name: drivers }
spec:
  backoffLimit: 0
  suspend: true                 # flipped to false by Kueue at admission
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: nextflow-runner
      containers:
        - name: driver
          image: …/advaita/nf-driver:latest
          command: [nextflow]
          args: [ …as Stage D… ]
          resources: { requests: { cpu: 500m, memory: 1Gi } }
          env:
            - { name: FOXFIRE_URL,    value: "http://foxfire.vivarta-echo.svc.cluster.local:9311/events/signed" }
            - { name: FOXFIRE_KEY_ID, value: nf-foxfire }
            - { name: RUN_NAME,       value: echo-20260717144738-d94b94 }
            - { name: NF_NAMESPACE,   value: vivarta-workers }
            - { name: NF_SERVICE_ACCOUNT, value: nextflow-runner }
            - { name: NF_WORK_PVC,    value: nextflow-work }
            - { name: NXF_CACHE_DIR,  value: /nxf-cache }
            - { name: FOXFIRE_CALLBACK_URLS, value: "http://echo-backend.vivarta-echo.svc.cluster.local:8080/callbacks" }
            - { name: FOXFIRE_SIGNING_KEY, valueFrom: { secretKeyRef: { name: foxfire-signing-key, key: pem } } }
          volumeMounts:
            - { name: wisp-overlay, mountPath: /etc/wisp }
            - { name: nxf-cache,    mountPath: /nxf-cache }
            - { name: vol0,         mountPath: /work }
      volumes:
        - { name: wisp-overlay, configMap: { name: echo-20260717144738-d94b94-overlay } }
        - { name: nxf-cache, persistentVolumeClaim: { claimName: nextflow-cache } }
        - { name: vol0, persistentVolumeClaim: { claimName: nextflow-work } }

FOXFIRE_CALLBACK_URLS is live on this run: echo-backend's /callbacks inbox receives the same signed events foxfire does — the callback fan-out of Stage G running in production, and the pattern a DataHub inbox subscribes with.

F — the queue, twice

Kueue admits the driver from drivers (its Workload object: job-echo-20260717144738-d94b94-f86ba, Admitted=True; the Job's suspend: true flips to false, condition "JobResumed"). With the driver image warm on the node, the first task pods appear 9 seconds after the Job — against ~6 minutes for the sibling's cold image pull the day before. Task pods are admitted from workers-immediate-cpu with zero queue wait.

consumes:  the driver Job, then 2 task pods (queue-name labels)
produces:  admission; node placement [K]
transport: Kueue drivers-cq / workers-immediate-cq

# from placement [F]: what admission produced
FIBONACCI: { pod: nf-ab9f8cd7…-b7df8, queue: workers-immediate-cpu, queue_wait_ms: 0 }
FIZZBUZZ:  { pod: nf-3e3a9ced…-c962b, queue: workers-immediate-cpu, queue_wait_ms: 0 }

A task pod's full label anatomy, as observed on the cluster — the overlay's three labels landing next to Nextflow's own and Kueue's:

labels:
  kueue.x-k8s.io/queue-name: workers-immediate-cpu   # ← overlay resourceLabels (gates admission)
  run-id: echo-20260717144738-d94b94                 # ← overlay (reaper join key)
  process: FIBONACCI                                 # ← overlay (reaper join key)
  kueue.x-k8s.io/managed: "true"                     # ← Kueue (plain-pod integration)
  kueue.x-k8s.io/podset: main                        # ← Kueue
  nextflow.io/app: nextflow                          # ← Nextflow's own set
  nextflow.io/processName: FIBONACCI
  nextflow.io/runName: echo-20260717144738-d94b94
  nextflow.io/sessionId: uuid-fee93b5e-…
  nextflow.io/taskName: FIBONACCI_attempt-1

G — the three ledgers, as fetched

consumes:  the run's lifecycle
produces:  plan · placement · trace in foxfire, joined on run_id + process
transport: POST /plan (wisp) · reaper pod watch · EC-P256-signed events
           (nf-foxfire) + callback fan-out (echo-backend /callbacks)

plan — what wisp decided: Stage C's per_process verbatim, plus the headline (sm/cpu/workers-immediate-cpu) and the driver block (image: nf-driver, namespace: vivarta-drivers, queue: drivers).

placement — what the reaper witnessed:

FIBONACCI:
  node: gf2a9c4                # nodepool barad-dur — an rtxp6000-8x GPU box (shared dev pool)
  flavor: on-demand            # capacity_type of the actual node
  image_id: …/nf-task@sha256:362e8f27…   # the OBSERVED digest, not the :latest tag
  timings: { created: 14:47:48, scheduled: 14:47:48, started: 14:47:50, finished: 14:47:55 }
  preempted: false
FIZZBUZZ:
  node: gf2a9c4
  flavor: on-demand
  timings: { created: 14:47:48, scheduled: 14:47:48, started: 14:47:51, finished: 14:47:55 }
  preempted: false

trace — what nf-foxfire measured, one signed event per lifecycle step (started, per-task submit/start/complete, completed; each carries a signature + trace_hash under key nf-foxfire):

FIBONACCI: { realtime_ms: 3454, peak_rss_mb: 45.1, exit: 0,
             cost: { total_usd: 0.0000382 } }
FIZZBUZZ:  { realtime_ms: 2460, peak_rss_mb: 44.9, exit: 0,
             cost: { total_usd: 0.0000272 } }
run: { status: succeeded, duration_ms: 15509, exit_status: 0,
       session_id: fee93b5e-3fd3-45e5-bb84-e09e5d436b6b, total_cost_usd: 0.0000654 }

On this run the plan and the placement agree — immediate work plans on-demand and lands on-demand. The deferred sibling is where the join earns its keep: its plan recorded flavor: spot (both processes derived spot-eligible), while the reaper witnessed flavor: on-demand — dev has no spot pool, so the deferred queue's spot-first quota admits the work and the pods land on on-demand hardware. Wisp's books and the reaper's witness disagree, the drift view surfaces it, and neither record overwrites the other. The trace meanwhile confirms the sizing on both runs: the envelope predicted ~45MB / ~3.4s p95; this run used 45.1MB / 3.45s — and its numbers are already inside the envelope that sizes the next submit. Its session_id is the resume lineage a future resume_of: echo-20260717144738-d94b94 resolves.

Overall flow

flowchart TD
  client["Client app<br/>(img-website / echo-frontend / DataHub)"]
  subgraph BE["wisp · Wanderland engine"]
    submit["submit<br/>[C]+[I] → context"]
    envelope["foxfire_envelope<br/>http adapter · fixture mock offline"]
    profile["profile<br/>stats [F] × class [Y] → per-process size/queue/spot [P]"]
    tmpl["classify<br/>when: match → ERB hydrate + overlay + resume [F]"]
  end
  foxfire[("Foxfire<br/>plan · placement · trace<br/>the authoritative ledgers")]
  reaper["reaper<br/>pod watch"]

  client -->|"POST /submit<br/>job_type, immediacy, params,<br/>resume_of, callback_url"| submit
  submit --> envelope
  foxfire -. "envelope [F] · resume lineage [F]" .-> envelope
  envelope --> profile
  profile --> tmpl
  tmpl -->|"plan [P]"| foxfire

  tmpl -->|"Job + overlay ConfigMap<br/>queue-name: drivers"| kDrv{"Kueue · drivers queue<br/>on-demand · caps workflows"}
  kDrv --> driverJob["Driver Job · ns vivarta-drivers<br/>nextflow -c base -c overlay"]
  driverJob -->|"task pods stamped from overlay<br/>queue-name + run-id + process [P]/[I]"| kWrk{"Kueue · workers queues<br/>spot-first, spill on-demand"}
  kWrk --> tasks["task pods · ns vivarta-workers"]
  tasks -->|"nf-foxfire signs each event"| foxfire
  tasks -.-> reaper
  reaper -->|"placement: node, node_class, gpus"| foxfire
  driverJob -->|"same signed events<br/>FOXFIRE_CALLBACK_URLS"| inbox["callback consumers<br/>(app inboxes)"]

  ui["UI"] -->|"reads the fold only"| foxfire

Per-process queue-name labels reach the task pods through the Nextflow overlay merge; the two Kueue admissions sit in separate cohorts, so admitting a driver never starves its own tasks.

Touch points, for later