ADV-669 · working design, in discussion · Graeme + Claude, building on Tim's proposal draft (2026-08-07)
This node is where we argue the design out. It builds on advaita-nextflow-router-data-model (the router this publishes into) and advaita-datahub-identity-and-storage-design (the app that consumes the catalog). Tim's proposal draft framed the goal and the holes; this node takes the position on the two decisions that actually shape the build — how the pipeline reaches the runner (the image fork) and where a registered template lives (the storage fork) — and scopes a Phase 1 that takes one honest step, not the whole journey.
The end state: a scientist defines an nf-core pipeline, publishes it, and it appears in DataHub's Run Workflow catalog — no DataHub release, no wisp rebuild, no engineering commit. Today adding a pipeline is a whole project: a driver repo, a Dockerfile, a hand-written wrapper main.nf, hand-built channels, a Jenkins job, an image build, and two Kotlin changes in DataHub. Every one of those is a place a scientist needs an engineer.
Phase 1 does not chase the end state. It removes the end-to-end. The test for anything in scope: does it shrink what the scientist has to do from "stand up a driver project" to "fill in a form that looks like a wisp template"? If yes, it's Phase 1. If it only matters for the customer-published / zero-review end state (per-tenant scoping, dropping the review gate, arbitrary-image sandboxing), it's deliberately deferred and named as such below. We are winning if the scientist's job gets small, even while engineering review stays.
This is the fork Tim's draft left open ("driver library architecture stays open") and it's upstream of almost every other question. Get it right and the storage question, the security question, and the tutorial all get smaller.
A driver image (fastqc-driver, demux-driver, cellranger-driver) bakes two very different things — and this is confirmed by nf-fastqc-driver/Dockerfile, which says it in as many words: "All the machinery — Nextflow, the nf-k8s executor, nf-amazon (S3), nf-foxfire (signed trace), and nf-advaita-datahub (the hub:// path factory + DataHub commit observer) — lives in nf-driver-base. This image is FROM the base + the workflow, which is all a driver image is."
the driver image = FROM nf-driver-base # the PLATFORM half — the plugins, shared
+ COPY workflows/<pipeline> # the PIPELINE half — the only per-pipeline bit
Only the second forces a per-pipeline build. The first — nf-driver-base (…/advaita/nf-driver-base:latest) — is genuinely shared platform code, and it's the real reason a base image exists: it's how our trace observer and DataHub plugins get onto the runner. The pipeline being COPY'd in next to them is an accident of convenience, not a requirement.
Restated the way Graeme put it: I don't need to build an image just to run a workflow. I should be able to submit the workflow down to an image. The base image is load-bearing for the plugins; the COPY is the ceremony we can drop by delivering the workflow at run time instead.
We have already shipped a workflow to a runner without building an image for it — echo's designer, and I checked the code (heading kept for continuity; the delivery is an env var, not a PVC — corrected here). The path is echo-frontend/js/designer.js → Echo::DagNextflow.render → Echo::Boundaries::SubmitDesign (vivarta-examples/echo-backend):
DagNextflow.render compiles a designer DAG into a runnable Nextflow DSL2 script — one process per node.SubmitDesign base64's that generated script into an env var; the driver decodes it to /tmp/main.nf and runs it against the base image's baked nextflow.config (k8s executor + nf-task container + nf-foxfire signing). Its own docstring: "Reuses the nf-driver image and the run infra… the only difference from launch_run is the workflow is generated, not baked."That is the existence proof for position B: the workflow arrives as data at run time and executes in the shared driver image, with no per-workflow image build. The nextflow pull nf-core/<x> route is a different delivery mechanism (git fetch from the nf-core registry vs a base64 env var) but the same principle — and echo proves the principle is already running in production.
A — baked per-pipeline image (today):
each pipeline is a repo → Dockerfile FROM nf-driver-base → bakes the workflow →
jcasc/vivarta builds + pushes to ECR → the template names that image.
cost: a repo + Dockerfile + Jenkins job + image build PER pipeline.
benefit: reproducible, offline, digest-pinned.
B — one base image, pipeline arrives at run time (end state):
nf-driver-base carries ONLY the plugins. the pipeline is pulled
(`nextflow pull nf-core/rnaseq -r 3.26.0`) or shipped as a compiled DAG onto a
PVC (the frontend precedent). NO per-pipeline image at all.
"publishing" collapses to pure metadata: pipeline id + version + exposed params.
Phase 1 — sits between them, on purpose:
KEEP the shared base image (it carries the plugins — Graeme: "if we need an
upstream image, we need to bake our configuration into it. that still exists").
but the PIPELINE stops being a baked path and becomes a VALUE in the registered
template — the coordinates `nf-core/<name> -r <version>` the base image pulls and
runs stock. no per-pipeline repo, Dockerfile, Jenkins job, or ECR image.
Phase 1 is the step toward B that pays off immediately: the scientist never builds an image, because the image they'd have built carried nothing but the platform plugins — which the base image already has. The workflow is nextflow pull-ed (or DAG-shipped) at run time. What the scientist supplies is the reference to the pipeline and the params to expose, not a container.
This also quietly resolves the security worry from Tim's draft §6.3 (see Fork 3): if the base image is fixed and platform-owned, the scientist is not choosing an image, an env secret, or an identity-token audience — there's simply no field for those in what they author. The dangerous half of the template stays engineering's, not because we fence it, but because the scientist's document doesn't reach it.
Tim's draft made "durably persist a registered template" sound like a substantial new build ("durably is the load-bearing word"). It is smaller than that, because the engine already persists records — we're just not using it for templates yet.
Every route into wisp's template set reads a file: config_view folds TemplateLibrary.overlay_templates(dir) off a ConfigMap dir-mount (config_view.rb:52), TemplateMatching#templates reads base + disk overlays on every classify (template_library.rb:14), and /templates/reload just busts the mtime memo and re-globs the disk. The ConfigMap path (k8s/templates-cm/* baked from config/<env>/templates.<name>.yml, applied + reloaded by vivarta.yml's /reload-templates route) is there because it was the easy way to get a reviewable file onto the pod — not because templates must be files.
Read the template from the records table instead of the ConfigMap. Same storage the engine already runs — verified in wanderland-core, not aspirational:
wanderland/storage/ ships sqlite, postgres, file, and null drivers, each exposing append(record) + overlay(addr) + a query surface (where, by_json, typed, under). boot_mount_storage.rb mounts a store per address-prefix from the storage: mounts: config — exactly the :pki: / :identities: / :trace: mounts wisp already declares (config.yml). A template store is a new durable mount (:templates: on the postgres driver), not a new datastore.Storage::Entity.upsert — this is Graeme's "capture deltas and upsert" verbatim. wanderland-core/lib/wanderland/storage/entity.rb is a diff-before-write upsert: it writes the seed when the address is absent, else appends a $merge poke for only the keys whose value changed, else does nothing — returning "created" | "updated" | "unchanged". So a re-publish of an unchanged template is a no-op; a changed one is a single delta. The poke/overlay model is the mechanism, already shared by the dependency graph and the event store.GET /templates and the matcher change source, not shape. Both read paths (config_view.rb:52 and TemplateMatching#templates, template_library.rb:14) point at the store instead of the disk glob; the base+overlay concat and first-match-wins precedence carry over. Precedence of a registered template vs a baked-base one needs a defined rule (registered shadows base, or the reverse) — a small decision, not a redesign.Entity.upsert append. The ConfigMap-reload objection ("a reload re-reads a file something had to commit") dissolves because there's no file.This is the reframe, now on firm ground: "durably persist" isn't a new datastore, it's a :templates: mount + Entity.upsert over storage the engine already runs. The ConfigMap stays as the engineer-authored base layer (platform templates); registered templates are the same shape, sourced from the table.
The whole point is that what a scientist fills in looks like a wisp template — the shape they'd see in GET /templates — but pared to only what they actually know. Concretely, the Phase 1 registration document is three fields:
# what a scientist registers — a template.yaml-shaped document, three fields
id: rnaseq # scoped to the tenancy tuple: key is org_id/team_id/owner_id/id
pipeline: # replaces a baked driver.image + workflow_path
nf_core: nf-core/rnaseq # the pull coordinates the shared base image runs stock
revision: "3.26.0"
parameters: # the offer/fence — an nf-core nextflow_schema.json subset
"$schema": "http://json-schema.org/draft-07/schema#"
type: object
properties:
aligner: { type: string, enum: [star_salmon, star_rsem], default: star_salmon }
genome: { type: string, description: "iGenomes reference key" }
required: [genome]
What the scientist does NOT write:
resources and task_processes are gone from the authored document. The profiler cold-starts every run from a safe class default and learns the real per-process CPU/memory from the first run's trace (the envelope → advaita-nextflow-router-data-model Stage C). Sizing is discovered, not declared — asking a scientist for min/max memory bounds is asking them to do the profiler's job.security_context, mounts — none are in this document. Engineering owns the mapping from pipeline: to the full run spec wisp launches (Fork 1 + Fork 3). The scientist writes coordinates + params; the platform supplies the rest.The full walk of how these three fields expand into a template and hydrate into a launch — where every field lands across the existing flow — is advaita-declarative-pipeline-publishing-data-model.
The record key is the whole tenancy tuple + id: (org_id, team_id, owner_id, id). This is Graeme's call — use the same org_id/team_id/owner_id triple that drive_file and pipeline_job already carry (advaita-datahub-storage-data-model) as the scoping key here too, so publishing never grows a second tenancy scheme and we can scope all the way down to a single user's pipeline without a later migration. We add the key now (nearly free on a greenfield record) and defer only the visibility enforcement — a filter over an already-keyed record, not a schema change.
Outstanding — who supplies the tuple at publish (unassigned). A scientist doesn't know their org/team/owner ids; those live in DataHub's authenticated context, not in a field they'd type. So the tuple must be stamped for them at publish time — most likely by the publish job, from the author's authenticated session, the same way the job already carries the wisp URL + creds. Confirming that (and who owns it) rides with §6.4 "who can run the publish job," because whatever runs it is what knows the author's identity.
Tim's draft §6.3 correctly notes that removing the commit removes the human review gate, and that a registered template is close to a full Pod spec (image, secret_env, identity_token, security_context, buildkit all ride template["driver"] through classify → JobManifest). At the customer / zero-review end state that's a real privilege-escalation surface.
For Phase 1 it is explicitly not our problem, by choice. The people registering are our scientists, working in our repositories, and their registrations are reviewed by engineers — the same trust anchor as today ("only wisp creates driver Jobs, and every template was reviewed"). Phase 1 keeps that gate. What Phase 1 changes is the work on the scientist's side of the gate, not the gate itself.
Two things make the eventual removal of the gate cheap, and Phase 1 gets both as side effects:
pipeline: + parameters: can't set an image or a secret ref because those aren't in the form. The escalation surface is small precisely because Fork 1 fixed the base image.So: Phase 1 carries the full-template risk on the review gate, knowingly, and the design leans toward a narrow scientist-facing document so that dropping the gate later is a small step rather than a security project.
Independent of every fork, and true today: an exposed param never reaches a stock pipeline. ParamsDoc.build (params_doc.rb:19) nests the caller's knobs under the params key of the document that is Nextflow's params (-params-file at classify.rb:126), so aligner lands at params.params.aligner while stock nf-core reads params.aligner. The three current drivers dodge it only because they're hand-written wrappers reading the sibling params.data.* map.
Phase 1 needs the flat form — a template opt-in (params_style: flat) that hoists the knobs to top level. The natural home is classify, not hydrate: hydrate builds the params doc before any template match (by design — it knows no job type), so it can't see a per-template flag. classify already holds both the built params_json and the matched template, so it's the place to hoist params to top level (guarding the reserved run_id/job_type/immediacy/data/tenancy keys), leaving data and tenancy where they are. Note this couples to the fence: flat mode puts caller values at exactly the top-level nf-core params that splice into ext.args → .command.sh, so params_style: flat and per-pipeline value validation ship together.
Also verified against the code, correcting Tim's draft §8: data does not tolerate an array today. hydrate.rb:88 rejects any non-String data value ("data values must be non-empty URI strings") before rendering ever happens — so if a pipeline needs list-valued inputs, the line to change is hydrate's validator, not template rendering. (params knobs are not scalar-checked there, so arrays survive in params but not data.)
Small, and mostly in wisp:
wisp:
- read templates from the records table (both read paths); ConfigMap stays as the
engineer-authored base layer.
- a registration upsert (poke/walkup delta → records table). no file, no commit.
- a `pipeline: { nf_core, revision }` field the base image runs stock (Fork 1, Phase 1 cut).
- `params_style: flat` in classify (the params.params fix) + per-pipeline value validation.
base image:
- nf-driver-base carries the plugins + `nextflow pull` at run (already most of the way there).
datahub:
- read the catalog from `GET /templates` (WorkflowRegistry already has reject-and-keep-previous).
- the Run Workflow catalog + input picker (frontend, not started).
deferred, named:
- per-tenant / customer scoping (org_id on the template record — cheap to add now, enforce later).
- dropping the review gate (Fork 3 — measured, not designed).
- full position B (DAG-to-PVC for non-nf-core / custom workflows — the frontend already shows it).
See advaita-declarative-pipeline-publishing-tutorial for the step-by-step a developer walks. In outline:
id: rnaseq, pipeline: nf-core/rnaseq @ 3.26.0, a two-param parameters block).POST /register); wisp validates the parameters block against nf-core/rnaseq's own nextflow_schema.json, then upserts the template record.GET /templates refresh sees rnaseq, fences the offer, and the catalog shows it.genome + aligner; DataHub POST /submits job_type: rnaseq with those as params.nextflow run nf-core/rnaseq -r 3.26.0 stock with params.aligner / params.genome at top level (via params_style: flat). No pipeline-specific image was ever built.The tutorial is the real test: if walking it feels heavy to us, it'll feel heavy to a scientist, and we cut until it doesn't.