Companion to advaita-declarative-pipeline-publishing (the design/forks) and its advaita-declarative-pipeline-publishing-tutorial (the walkthrough). Written in the ledger idiom of advaita-nextflow-router-data-model — where each field comes from, what carries it, what consumes it — extended to answer the one build question: do we hydrate at runtime, or bake a project into images?
This node exists to make the data visible: a scientist writes three fields; those three fields, plus what the platform supplies, become a full wisp template; that template, plus what a user supplies at run time, becomes a Kubernetes launch. Nothing here invents a new channel — it names where the scientist's data enters the flow the router and storage models already run.
Graeme's question was the fork: do we hydrate everything at runtime, or hydrate the three-field doc into a project template that then builds driver images? The answer this design takes:
PUBLISH TIME (once per pipeline): EXPAND the 3-field doc → a full template RECORD.
a pure DATA transform. no image, no build, no Docker.
SUBMIT TIME (every run): HYDRATE the record → a Job, through the EXISTING
router chain. the shared base image runs the pipeline
stock. nothing is compiled in the loop.
There is no build-time project template that produces images. That was the third option (position A in the design's Fork 1) and we rejected it: baking a driver image per pipeline is the very ceremony we're removing. Because the base image is shared and pulls the pipeline at run (Fork 1, Phase 1 cut), publishing never touches a registry. "Publish" is an upsert of a data record; "run" is the router chain that already exists.
This is the whole reason the model is cheap: the only genuinely new thing is a data expansion, and the store it lands in. Every launch mechanic is unchanged.
The router page tags each field by where it comes from. Declarative publishing adds two sources at the front — the two authors of a template that used to be one hand-written file:
| Tag | Source | When |
|---|---|---|
| [S] | Scientist — the 3-field registration doc (id, pipeline, parameters) |
publish time |
| [O] | Operator/platform — the mapping that fills the run-spec half of the template | publish time |
| [C] | Client — the caller's submit (DataHub): job_type, params values, data, tenancy |
request time |
| [I] | Identity — the minted run_id |
request time |
| [F] | Foxfire — the per-process envelope (sizing history) | classify time |
| [P] | Profiler — sizes computed over [F] within the class bounds | classify time |
| [Y] | Template — the stored record, now = [S] ∪ [O] instead of a baked file | boot/registration |
| [K] | Kueue — placement injected at admission | admission time |
The one structural change to the router model: [Y] used to be a file an engineer hand-wrote. Now [Y] is a record, and it has two authors — [S] the scientist (a narrow subset) and [O] the platform (everything dangerous or tedious). Everything downstream of [Y] is byte-for-byte the existing flow.
This is the step that's new. A registration mapping takes the curator's [S] document — a curator, not the end scientist: the product is turning a pipeline's ~133 params into the ~6 a biologist should see (ADV-670 item 5) — and merges the platform's [O] fields into the same template shape classify/profile/validate_params already consume. The platform writes exactly the fields the curator omits.
# [S] — what the CURATOR registered (the whole authored doc; ADV-670 shape)
id: rnaseq-star-salmon
pipeline: # ONE coordinate, TWO forms (ADV-670 item 4):
nf_core: nf-core/rnaseq # a direct nf-core pull …
# repo: advaita-corporation/nf-demultiplex # … OR a fork (demux needs one)
revision: "3.26.0" # curator types a tag; [O] pins it to a SHA (below)
size: large # a t-shirt HINT (item 3), not min/max bounds
plugins: [nf-schema@2 # pipeline plugin pins the driver must load
parameters: # scalar knobs AND file inputs (item 2)
properties:
input: # a FILE input — the user picks it
{ type: string, format: file-path, pattern: "\\.csv$" }
genome: { type: string, enum: [GRCh38, GRCm39] } # curator narrowed a free-text field
aligner: { const: star_salmon } # curator PINS + hides it — the point of the reg
required: [input, genome]
# [O] — what the PLATFORM mapping adds at registration (never curator-set)
when: { job_type: rnaseq-star-salmon } # from id (org-scoped key; see Tenancy)
driver:
image: …/advaita/nf-driver-base:latest # the SHARED base — VERIFIED: nf-driver-base bakes
# Nextflow + nf-k8s + nf-amazon + nf-foxfire +
# nf-advaita-datahub (nf-fastqc-driver/Dockerfile)
command: [nextflow]
run_opts: [run, nf-core/rnaseq, -r, <resolved-sha>, -profile, kubernetes, -name, "<%%= run_id %>"]
identity_token: { audience: datahub }
env: { DATAHUB_URL: …, AWS_S3_ENDPOINT: … }
# service_account is NOT per-org today — one NF_SERVICE_ACCOUNT (Tenancy: designed, not built)
resources: <expanded from `size: large`> # t-shirt → a resource class; profiler refines from trace
params_style: flat # hoist params to top level for stock nf-core
# [Y] — the stored record = [S] ∪ [O], keyed by (org_id, team_id, owner_id, id)
Four things to see:
pipeline is one coordinate, two forms — nf_core: (direct pull) or repo: (a fork). Demux runs a fork because upstream's output{} DSL can't run on the Nextflow our SSE-C nf-amazon fork pins. The base image runs whichever, stock. No image is built for it.revision is SHA-pinned by [O]. The curator types a human tag (3.26.0); the mapping resolves it to a commit SHA at publish and stores/runs the SHA — because a git tag is a movable pointer (a maintainer can force-push it), and a cold pod would then pull different bytes for the same tag with no sign in the logs. The SHA is the only value that can't move. Curator convenience, platform reproducibility.parameters carries both kinds — scalar knobs (aligner, genome) and file inputs (format: file-path). DataHub renders the first as controls, the second as a Drive file picker. The curator's two levers are plain JSON Schema: const pins-and-hides a value (aligner: star_salmon is what makes this registration STAR+Salmon rather than pseudo-salmon), and narrowing an enum turns a free-text field into the few references we host.image, identity_token, env (secret refs), mounts are platform-authored; the curator's doc has no field that reaches them. Sizing is a single t-shirt size: — [O] expands it to a resource class, and the profiler refines per-process from the first trace.The record lands in the records table (design Fork 2): Entity.upsert writes it at addr = :templates:<org>/<team>/<owner>/<id>, type_addr = :types:wisp-template, payload = [Y]. Both read paths (config_view / TemplateMatching#templates) overlay it back — same shape they read from the ConfigMap today, different source.
Now a user runs it. DataHub submits [C]; the router chain hydrates the record [Y] plus the submit [C] into a Job. This is the router page's Stage A→F verbatim — the table below just marks where the scientist's data (via [Y]) and the user's data (via [C]) each land.
S0/A submit ─▶ B hydrate ─▶ validate_params ─▶ resolve_inputs ─▶ C profile ─▶ E classify ─▶ F Kueue ─▶ driver ─▶ tasks
[C]+[I] [C→doc] [C×Y] [C×Y→D→F] [F+Y→P] [Y+C hydrate] [K]
| Datum | From | Carried by | Lands at / consumed by |
|---|---|---|---|
id → when.job_type |
[S]→[Y] | the record | classify/profile match the submit's job_type to this record |
pipeline.nf_core/repo + revision |
[S]→[Y] | argv | nextflow run <coord> -r <sha> — the driver process, in the base image |
parameters (the schema) |
[S]→[Y] | the record | validate_params gate at submit [wisp]; GET /templates catalog form [DataHub] |
scalar params values (aligner, genome) |
[C] | params-file (params.json) |
params.aligner/params.genome — top level via params_style: flat |
| file inputs (paths) + file-sets | [C] | params (paths) / data (staging) | expanded → materialised to <root>/<path> before nextflow (see below) |
tenancy tuple |
[C] | Job metadata + params-file | org→SA at render (designed); team/owner→Allocate ACL; stamped on the plan |
base image, env, identity_token, mounts |
[O]→[Y] | Job spec / driver env / token file | the driver pod; DataHub TokenReview (ROLE_DRIVER) |
resources class + total_bytes |
[O]→[Y] / [D] | consumed by profile / classify |
per-process size [P]; the per-run work volume sized from total_bytes |
run_id |
[I] | argv -name, Job name, labels |
the join key for plan/placement/trace (advaita-foxfire-events-data-model) |
| per-process sizes | [F]+[Y]→[P] | overlay ConfigMap | Nextflow config merge → each task pod's requests + queue-name label |
| placement | [K] | injected at admission | node/flavor; recorded by the reaper |
Read the table as the answer to "some of it becomes a wisp template, some goes here, some goes there": the schema half of parameters is a template concern (validation + catalog); the value half is a per-run params-file concern; pipeline is argv; tenancy is identity + ACL + the plan; sizing is the profiler + overlay. One small document fans into exactly the router channels that already exist.
The file inputs a user picks are materialised at their DataHub paths onto the per-run work volume before nextflow starts, so a stock nf-core samplesheet runs verbatim — nf-schema stats ordinary local files. A new resolve_inputs step (after validate_params, before classify) expands the selection owner-fenced into an opaque plan (paths + fileIds + total_bytes, no keys), records a signed input_plan to foxfire, and sizes work-<run_id> from total_bytes; the driver then batch-resolves the SSE-C keys run-bound and streams each object to <root>/<path>. The full ledger — the two-resolution security spine (keys only at execution), the layout map, and the driver utility — is advaita-datahub-input-materialisation.
This supersedes the earlier lazy design (resolving hub:// refs in the samplesheet via HubPathFactory.parseUri at file() time, which forced a .localized.csv cell-rewrite and could not glob). HubPathFactory's write path still publishes outputs (advaita-datahub-storage-data-model S4) — only the input read path is retired.
params_style: flat is in the expansionThe one place the existing flow does not already work for a stock pipeline: hydrate nests the user's knobs under params (params.params.aligner), but stock nf-core/rnaseq reads params.aligner. So the expansion sets params_style: flat on the record, and classify (which holds both the built params.json and the matched record) hoists the knobs to top level before mounting the params-file — guarding the reserved run_id/job_type/immediacy/data/tenancy keys, leaving data and tenancy where they are. hydrate can't do this (it runs before the match, so it can't see params_style); classify is the home. This is the design node's §7, shown here as the single transform that makes the stock pipeline read its params.
Graeme's call: scope by the full tenancy triple — org_id, team_id, owner_id — everywhere, because it's the tenancy tuple for everything else already (advaita-datahub-storage-data-model confirms drive_file and pipeline_job carry exactly this triple; object layout is <orgId>/<teamId>/<ownerId>/<fileId>). Using the same triple here means the publishing model never has to grow a second scoping scheme.
Where the tuple lives, at each time:
PUBLISH: the record KEY is (org_id, team_id, owner_id, id) — so two orgs (or teams, or users)
can publish `rnaseq` without collision. visibility ENFORCEMENT is deferred; the KEY is
present from day one, so enabling enforcement later is a filter, not a migration.
SUBMIT: the submit [C] carries `tenancy: {org_id, team_id, owner_id}`, asserted by DataHub
(the authenticated decision point) — router Stage A / storage S0.
RENDER: org_id → the per-org runner ServiceAccount. DESIGNED, NOT BUILT: today wisp launches
every run as ONE service account (`NF_SERVICE_ACCOUNT`, default nextflow-runner —
classify.rb:152 / config.yml:466); there is no org→SA registry yet. Per-org SA
resolution is the storage model's design (its Tenancy section) and remains future work.
RUNTIME: team_id + owner_id are enforced by DataHub's `input:allocate` (the per-file SSE-C key is
the capability) — never by the OIDC token. storage model "Tenancy" section.
Outstanding — who owns injecting the tuple, unassigned. A developer/scientist authoring a pipeline does not know org/team/owner ids — that's DataHub's authenticated context, not something they type. So the tuple has to be supplied for them, and by whom is open:
id is auto-scoped), or does the curator pass nothing and the platform derive it? Leaning: the publish job derives it, same as it wires the URL + creds — the curator never sees a tenancy field.The publish-time owner is the open item. It belongs with the design's §6.4 "who runs the publish job," because whatever runs it is what knows the author's identity.
Almost nothing moves. The unchanged spine is the entire value:
unchanged (reused verbatim):
- the whole submit → hydrate → validate_params → profile → classify chain (router A–F)
- the params-file / overlay / Job-metadata / env / volumes transports
- profile's sizing, sizes.yml/queues.yml/thresholds.yml, Kueue admission
- the storage round-trip (creds → allocate → put → commit) and the foxfire ledgers
- GET /templates as the catalog source; WorkflowRegistry's reject-and-keep-previous
new (the only new parts):
- the EXPANSION mapping [S] ∪ [O] → [Y] (publish-time data transform, no build)
- the RECORDS-TABLE source for [Y] (both read paths point at it; ConfigMap stays as the
engineer-authored base layer)
- `pipeline: {nf_core, revision}` as a run target the base image runs stock (vs a baked path)
- `params_style: flat` in classify (the params.params hop)
- the (org,team,owner,id) record key
The base image is the one pre-built artifact, and it's platform-owned and shared — built when the plugins change, not when a pipeline is published. That's the line that keeps "publish" a data operation.
rnaseq.yml; the mapping expands it to the full record above and upserts it under (advaita, proteomics, u-31f2, rnaseq).GET /templates refresh sees rnaseq, renders parameters as the form.genome: GRCh38; DataHub POST /submits {job_type: rnaseq, params: {genome, aligner}, data: {reads}, tenancy: {advaita, proteomics, u-31f2}}.validate_params checks genome/aligner against the schema; profile cold-sizes from the class; classify matches the record, resolves advaita→runner SA, hoists params to top level, and applies the Job.nextflow run nf-core/rnaseq -r 3.26.0 -profile kubernetes with params.genome=GRCh38 at the top level. Task pods size from the overlay; outputs go through allocate/commit; foxfire records plan/trace. No rnaseq image ever existed, and nothing was compiled at publish.2026-08-13 — a design-vs-code pass. The expansion + hydration are implemented; two example details in this doc trail the code, and one platform-half wiring is still open.
driver.command is [nf-driver-entrypoint], not [nextflow]. The [O] example above shows command: [nextflow]; the code (template_expand.rb) now emits [nf-driver-entrypoint] — the materialise-then-nextflow wrapper baked into nf-driver-base (advaita-datahub-input-materialisation "Built"). It runs the input materialiser fail-closed, then exec nextflow "$@". It has to be the container command (not just the image ENTRYPOINT) because a k8s command overrides ENTRYPOINT, so wisp sets it on the [Y] driver; a non-DataHub run passes straight through to nextflow.
The hydration chain is async, not the synchronous diagram. The Stage A→F diagram reads as one synchronous chain; the code splits it: POST /submit runs submit → hydrate → validate_params → enqueue_request and returns 202, releasing the request thread; a wisp materialise worker then drains the record and runs the tail foxfire_envelope → profile → resolve_inputs → classify (which launches the driver). Same stages, same order — just off the request thread. The async truth (queue genesis :events:request, terminal :events:request:completed) is in advaita-datahub-input-materialisation's Sequence.
The one open wiring — [O] env/mounts. This doc says the platform ([O]) adds env/mounts, and notes they "fall to TemplateExpand's defaults for now." In the code there is no hardcoded default: template_expand.rb reads driver.env/driver.mounts straight from config.publishing (pub["env"]/pub["mounts"]), and today config.publishing carries only base_image / config_path / identity_audience / pin_revisions / classify_params. So an [S]-registered template currently expands to a driver with an empty env — no DATAHUB_URL, no object-store endpoint, no NXF_OFFLINE override — and cannot reach the storage plane. The hand-written [Y] ConfigMap templates (fastqc/demux/cellranger) carry env inline, which is why they run and the registered path does not yet. Closing this is populating config.publishing.env/mounts (the platform storage-plane block) — the last step before a registered pipeline launches end to end.
Everything else in this model — /register, SHA-pinning, layer classification, the [S]→[Y] expansion, size→resources, params_style: flat / route-then-flatten, the (org,team,owner,id) record key — is built and matches the intent.