Data model · 2026-08-11 · the INPUT half of the DataHub ↔ driver handoff. The output half is advaita-datahub-storage-data-model; the launch chain is advaita-nextflow-router-data-model; the trust spine is advaita-datahub-identity-and-storage-design. Written in the router's ledger idiom. Supersedes the hub://-refs-in-a-samplesheet approach — inputs are now materialised at their DataHub paths before nextflow starts, so a stock, unmodified nf-core pipeline runs against ordinary local files.
nf-schema reads a samplesheet as text, regex-checks each cell, and stats the file — all before file() is ever called. The scientist's sheet cells are ordinary relative paths (reads/sample1_R1.fastq.gz), written on their laptop. So the only way to run the pipeline unmodified is to have real files at those relative paths at launch.
Materialise the scientist's files, at their DataHub paths, onto the per-run work volume, before
nextflowstarts. By launch there are only ordinary local files at ordinary local paths — the sheet works verbatim, and nothing in the pipeline needs to understandhub://.
This retires the lazy path-factory read (HubPathFactory.parseUri for inputs) and the .localized.csv cell-rewrite. The path factory's write half still publishes outputs (advaita-datahub-storage-data-model S4) — the plugin stays; only the input read path is dropped.
Everything secure here comes from splitting what looks like one step into two with different trust and different timing. Never conflate them.
| Expansion | Key resolution | |
|---|---|---|
| When | submit time | execution time (driver) |
| Who calls | wisp → DataHub | driver → DataHub |
| Produces | { target_path → opaque fileId, size } |
{ object_key, sse_c_key } per file |
| Key material | none | the SSE-C keys — transient, in the driver |
| Auth | wisp SA token; DataHub owner-fences by its OWN run record | driver run-bound token (pod-name ⊃ run_id) |
| Stored? | yes — opaque plan, at rest (DataHub + foxfire) | never — minted per call, held only in RAM |
The opaque plan is safe at rest: a fileId is meaningless without DataHub, an object_key alone fetches only SSE-C ciphertext. The SSE-C key — the actual capability — is minted per-object by DataHub and handed only to the run-bound driver, at materialisation, never written to any durable record. That is the invariant of advaita-datahub-identity-and-storage-design, preserved verbatim; this flow only batches and pre-stages around it.
| Tag | Source |
|---|---|
| [C] | Client — the scientist's selection, via DataHub's submit |
| [D] | DataHub — the expansion (owner-fenced) and, at exec, the keys |
| [Y] | Template — the curator's micro-template, incl. the layout map |
| [I] | Identity — the minted run_id (the join key + the pod-name bind) |
| [P] | wisp — the plan it records + the volume size it derives |
| [F] | Foxfire — the signed input_plan provenance record |
The curator declares where each picked file materialises — by annotating the file params themselves with a path:, rather than a separate layout block that would restate the parameter names. This keeps the target path next to the schema that validates the pick (one place), and makes an assumption explicit instead of hidden: only file/file-set params materialise, and each says where it lands.
# part of the curated [Y] document
parameters:
properties:
input: { type: string, format: file-path, path: "samplesheet.csv" }
reads: { type: string, format: file-set, path: "reads/" }
genome: { type: string, enum: [GRCh38, GRCm39] } # a scalar — NOT materialised
required: [input, reads]
Two things this makes precise:
file-set's path: is the target ROOT. The picked folder's contents materialise under it with their relative structure preserved — because the samplesheet's reads/sample1_R1.fastq.gz cells must resolve at exactly that path. The scientist's sheet and folder are consistent by construction (both written on their laptop); wisp roots the set at path: and keeps the tree.genome selects a reference the pipeline resolves itself (igenomes, or --fasta <url> the pipeline fetches) — references are params the pipeline pulls, never files this flow materialises. So for the Phase-1 pipelines (demo, rnaseq, demultiplex) every materialised file is a scientist pick, and the param-only assumption holds.wisp owns interpreting this (globs, recursion, path assembly). DataHub never learns the layout vocabulary — it only answers "list the files under this folder, owner-fenced." That ownership split is why wisp drives resolution (it holds the layout), rather than DataHub expanding before it calls /submit.
The escape hatch, when the assumption breaks (deferred). A curator constant that isn't a scientist pick — a private/custom reference, a fixed multiqc_config.yaml, a default sheet template — has a source but no param. Those get a separate assets: list ({ source: hub://<id> | https://…, path: "…" }), curator-sourced, never in the scientist schema. None of the Phase-1 three need it, so it stays an open item, not built.
layout [Y] ─▶ M0 submit ─▶ M1 resolve_inputs ─▶ M2 classify ─▶ M3 materialise ─▶ M4 nextflow
selection[C] plan[D] + input_plan[F] size[P] keys[D]→disk verbatim sheet
+ total_bytes ───────────────┘
DataHub authorises the user, then submits the scientist's selection as paths — the same S0 body (advaita-datahub-storage-data-model), with the two-map split doing real work:
{
"job_type": "rnaseq-star-salmon",
"tenancy": { "org_id": "79b03d…", "team_id": "f94cf1…", "owner_id": "bob" },
"params": { "input": "samplesheet.csv", "genome": "GRCh38", "outdir": "results" },
"data": { "reads": "reads/" }
}
params — pipeline-facing values, incl. file paths (input). Flattened to top-level by params_style: flat (wisp classify.flatten_params), because the pipeline reads params.input.data — staging-only selections the pipeline has no param for (reads: nf-core/rnaseq has no params.reads). Stays nested (params.data.reads) so it can never become a top-level pipeline param — exactly what leaving data unflattened achieves. It rides the submit only so validate_params can check it against the curated schema.What is not here: the FASTQs, or any key. A glob may match 500 files — that list is the plan (M1), kept off the body so key material never touches submit and the body stays fixed-size.
resolve_inputs (wisp → DataHub expansion; the new step)A step in the /submit chain after validate_params, before classify — because classify must size the volume from the result. wisp interprets the layout, and asks DataHub to expand each entry owner-fenced:
consumes: layout [Y], selection [C], run tenancy [C]
produces: the plan [D] (paths + opaque fileIds + sizes, NO keys); total_bytes [P]
transport: wisp → DataHub POST /runs/{run_id}/resolve (batch: send the layout+selection, get the plan)
wisp → foxfire /collect (the signed input_plan record)
Batch expansion — one call, not one-per-glob: wisp POSTs the layout + selection; DataHub expands every glob/dir and returns the whole plan.
{ "root": "/work/inputs", "total_bytes": 41243967,
"files": [
{ "path": "samplesheet.csv", "file_id": "2a49692a…", "size_bytes": 75 },
{ "path": "reads/sample1_R1.fastq.gz","file_id": "4db09e37…", "size_bytes": 41243892 }
] }
Auth, precisely. wisp presents its own projected SA token (aud: datahub) — new; today only the driver holds one. DataHub owner-fences by the tenancy on its OWN pipeline_run record, not by what wisp asserts (wisp echoing the tenancy is a hint, never the authority). And expansion authz is only belt — the suspenders is M3's per-file allocate check, which re-validates each grant when the key is actually handed out.
Two rugs, no keys. DataHub persists the plan as the authoritative per-run input ACL. wisp emits a signed, typed input_plan to foxfire — same path as the run plan (classify.emit_plan → FoxfireSigner.gate_body → /collect), joined on run_id, carrying { root, files:[{path, file_id, size}], total_bytes } and no keys. It is the run's input provenance (audit + reconcile), sibling to plan/placement/trace (advaita-foxfire-events-data-model).
total_bytesThe per-run work volume already exists (case task-7be1e663): classify.resolve_work_volume mints work-<run_id>, and adopt_work_volume PATCHes an ownerReference to the driver Job so Kubernetes GCs it — no cleanup handler, no reaper, no onComplete. WISP_WORK_ACCESS_MODE is ReadWriteOnce single-node / ReadWriteMany multi-node (CoreWeave shared-vast). The one delta: size from the plan's total_bytes (a submit/plan-carried hint) instead of the static WISP_WORK_SIZE, so a run that won't fit is rejected in seconds rather than dying Disk quota exceeded twenty minutes in.
A utility baked into nf-driver-base, run by the entrypoint before nextflow:
consumes: the plan (from DataHub, authoritative)
produces: the DataHub tree replicated under <root> as plaintext files
transport: driver → DataHub (batch: run token → {path, object_key, sse_c_key, size} for ALL files)
driver → CoreWeave (SSE-C GET per object, parallel)
Batch key resolution — one run-bound call returns every file's object_key + sse_c_key (the batched input:allocate), replacing N per-file calls. DataHub TokenReviews the driver token (ROLE_DRIVER, pod-name ⊃ run_id), re-checks each fileId against the run's tenancy ACL, 403s any the run may not read. The utility then writes each object to <root>/<path> with an SSE-C GET.
Note the materialiser needs its OWN SSE-C client — it runs before nextflow, so it cannot lean on the nf-amazon fork; an aws-cli --sse-customer-* GET or a small script threads the per-object key. This is the one genuinely new driver-side dependency.
nextflow -c … run nf-core/rnaseq -r <sha> -params-file … --outdir results
# launchDir = /work/inputs, so `reads/sample1_R1.fastq.gz` in the sheet resolves
Ordinary local files, real names, the samplesheet verbatim — nf-schema stats and passes. Outputs then publish via the write path (advaita-datahub-storage-data-model S4).
The flow is async. /submit persists the run's snapshot (:events:request) and returns 202 — the request thread is released, never blocked on DataHub (killing the re-entrant deadlock). A wisp materialise worker drains that record within seconds and runs the tail: resolve_inputs (→ :events:storage, the keyless storage plan that sizes the volume), classify (→ :events:plan, the compute placement — unchanged), then launches the driver Job and marks :events:request:completed. Two output plans, two axes, joined by run_id; the billing rollups keep reading :events:plan untouched.
The Kueue slot wait is the driver Job's, after launch — so the file-deletion window is between resolve (~submit time) and the driver's materialise an hour later. The driver's all-or-nothing 404, diffed against :events:storage, is the "was there at resolve, gone now" audit.
shape: sequence_diagram
sci: scientist
dh: DataHub
ws: wisp / submit
fox: foxfire events
wk: wisp materialise worker
drv: driver Job
cw: CoreWeave
nf: nextflow
sci -> dh: pick workflow, folder, samplesheet
dh -> dh: authorise · stamp its OWN pipeline_run
dh -> ws: submit flat picks, no keys
ws -> ws: validate_params · mint run_id
ws -> fox: persist :events:request (submit snapshot)
ws -> dh: 202 run_id — thread released {style.stroke-dash: 4}
dh -> sci: accepted — preparing {style.stroke-dash: 4}
wk -> wk: drain :events:request within seconds
wk -> dh: resolve_inputs — layout + selection, no tenancy
dh -> dh: expand owner-fenced by ITS OWN run record
dh -> wk: plan — paths, fileIds, sizes, total_bytes — NO keys {style.stroke-dash: 4}
wk -> fox: :events:storage (signed, keyless) — sizes work-run
wk -> fox: :events:plan (compute placement)
wk -> drv: launch Job — volume from storage tier / total_bytes
wk -> fox: :events:request:completed — queue terminal
drv -> drv: wait for a Kueue slot (the real wait)
drv -> dh: inputs/materialise — run-bound token, no body
dh -> drv: object_key + sse_c_key per file — all-or-nothing, 404 aborts {style.stroke-dash: 4}
drv -> cw: SSE-C GET per object, write root/path
drv -> nf: nextflow run — launchDir is the volume
nf -> nf: local files, samplesheet unmodified
nf -> dh: publish outputs — allocate, PUT, commit
dh -> sci: results in Drive {style.stroke-dash: 4}
The materialise worker's queue is :events:request cancelled by :events:request:completed at the same cell (the run-lifecycle sibling pattern), with a dead-letter on max retries:
stateDiagram-v2
[*] --> requested: submit writes the request, then 202
requested --> in_flight: worker pops (seconds)
in_flight --> completed: resolve, classify, launch OK
in_flight --> requested: transient failure, retry
in_flight --> dead_letter: max retries
completed --> [*]: request completed
dead_letter --> [*]: request failed
Removes: hub:// in samplesheets · the HubLocalizer/.localized.csv rewrite · any Nextflow input extension (Nextflow offers no download hook, so per-task remote fetch never existed) · the validate_params pattern conflict (a .csv path matches ^\S+\.csv$).
Buys: sizing before launch (total_bytes → the volume is derived, not guessed) · sheet validation at submit ("rows 3, 7 reference a file not in this folder" beats a mid-run schema error) · a named per-run input ACL (the plan authorises exactly what it named, owner-fenced).
Costs, honestly: eager materialisation downloads the whole selection up front (fine for nf-core, which processes the full sheet; a front-loaded startup for a 500-file glob) · a new sync resolve_inputs dependency in submit (DataHub-down → a clean 503, not a hang) · the plan is a snapshot — a file deleted before exec 403s at allocate (the live gate), so the driver fails clearly.
A reference is a named, registry-held bundle of inputs a run selects by name — the reference-genome analogue of a file-set. It differs from a scientist pick two ways: the members are fixed by the registry entry, not picked per run, and the entry carries both file members and scalar params. A reference passes through rather than materialising: each member is a param value the workflow's own provider resolves at task time, the sibling of the materialised pick this node otherwise describes. synth.happy_birthday consumes a reference set this way.
The reference registry is owned by wisp — one record per named set. wisp holds the write and read surface, so a consumer-facing DataHub carries no curator management UI.
references:
GRCh38-ensembl116:
title: "Human GRCh38 · Ensembl 116" # the picker label
shared: true # a public set any run may read
materialisation: lazy # the value stays on the param; nf-amazon fetches it
params: # role -> value (the names the workflow reads)
fasta: s3://ref-data/datahub/v1/genomes/homo_sapiens/GRCh38/genome.fa
gtf: s3://ref-data/datahub/v1/genomes/homo_sapiens/GRCh38/genes.gtf
star_index: s3://ref-data/datahub/v1/indices/star/2.7.11b/homo_sapiens/GRCh38
gencode: false # a scalar — passed through untouched
params is role → value. A scheme-bearing value (s3://) names a file the workflow reads as params.<role>; the workflow's own provider stages it. A bare string is a scalar the workflow reads verbatim. The record key is the workflow's param name.materialisation — lazy (the built path) leaves each s3:// value on its param, and the nf-amazon fork stages it into the task pod at first access. eager (copy the whole set to a work volume before nextflow) is not built.shared and tenancy — a shared set is public: any run's tenancy resolves it. A tenancy-scoped record fences a set to one org. A run resolves its own private set of a name first, else the first shared set of that name.A reference param renders a picker of named sets — the sibling of the file-path and file-set pickers:
parameters:
properties:
guest_set:
type: string
format: reference
enum: [happy-birthday-fairytale, happy-birthday-superhero]
default: happy-birthday-fairytale
The submit value is a set NAME. The set's members become top-level params keyed by their record roles, so each role must match the name the workflow reads — params.guests, params.adjectives. A reference param carries no target path: a member is a param value, not a file staged to a directory.
Because wisp owns the registry, hydration is a wisp step at classify — expand_references, before the driver Job launches. It is a pure params transform, not a storage-plan step.
expand_references reads the reference param value (the set name) and looks the record up in the wisp :references: store (ReferenceStore.resolve).params map into the run's params, first-writer-wins — a value the submit set explicitly is never overwritten, so a per-run override stays deliberate. ParamRouter.route then hoists each member to top-level params.<role>.s3:// value stays on its param, and the nf-amazon fork stages it into the task pod when a process reads file(params.<role>). A scalar rides untouched. A hub:// value rides to the DataHub path factory instead.A reference never enters the storage plan and mints no key — it does not touch the pick-materialisation path this node describes. A run may carry both: materialised picks and reference params, each resolved by its own route.
Ownership variant. A DataHub-owned registry would expand the reference during its own submit-time step instead. Only the expansion site moves; the launch params are identical.
shape: sequence_diagram
cur: curator
sci: scientist
ws: wisp
drv: driver Job
cw: CoreWeave
nf: nextflow
registration — once, ahead of any run: {
cur -> ws: register reference — params: role -> s3:// value or scalar
ws -> ws: store the reference record
}
sci -> ws: submit — a param carries reference = <name>
ws -> ws: classify · expand_references — members into params.<role>
ws -> drv: launch Job — params carry s3:// values + scalars
drv -> nf: nextflow run — file(params.guests) is s3://…
nf -> cw: nf-amazon GET per member at task start
nf -> nf: render outputs, then publish
aud: datahub token + a DataHub resolve endpoint + role. New: only the driver holds one today. DataHub must derive tenancy from its own run record.total_bytes rides — the submit body (DataHub pre-expands) vs the resolve_inputs response (wisp expands). This node assumes the latter (wisp owns the layout); revisit if the layout ever moves to DataHub.shared-vast, per-run mint on dev), so whether it works is settled; the startup latency it adds is the open spike.format: file-set in validate_params + the offering — a set's member glob must not land in JSON Schema pattern (regex-on-value collision); Offering.file_inputs must count file-set as a file input.2026-08-13 — the plan above is implemented; rebuilds through the nf-driver-deploy Jenkins job.
M3 driver materialiser — a Groovy shadow jar, not aws-cli. nf-advaita-datahub's :materialiser subproject builds datahub-materialiser.jar, a standalone app that uses the AWS SDK v2 directly for the SSE-C GETs (the same SDK the nf-amazon fork bundles; it runs BEFORE nextflow so it cannot use the in-process nf-amazon bridge — this supersedes the M3 "aws-cli or a small script" note). nf-driver-base COPYs the jar to /opt/datahub/ and swaps its entrypoint for nf-driver-entrypoint (materialise → cd <root> → exec nextflow "$@", set -e = fail-closed: an all-or-nothing failure aborts before nextflow). Because a k8s container command overrides the image ENTRYPOINT, wisp sets the driver command to nf-driver-entrypoint (template_expand + classify), so the materialiser fires on every driver launch; a non-DataHub run prints nothing and passes straight through to nextflow. DatahubClient.materialise() is the one batch call — POST /runs/{run}/inputs/materialise, no body → {root, total_bytes, files:[{path, object_key, sse_algorithm, sse_c_key_b64}]}.
Contract + sim. The real endpoints exist in advaita-drive-service @ eed2c21 (POST /runs/{id}/resolve ROLE_WISP keyless plan; POST …/inputs/materialise ROLE_DRIVER keyed, no body). sim-datahub mirrors both — its resolve + input-materialise boundaries consult the store-map as a first-class file source (tenancy + size + real object_key), so a curated run reads REAL Floci test data instead of minted keys.
Tests + gate. The materialiser carries a full suite — unit (SSE-C threading, fail-closed, all-or-nothing) + a real-Floci SSE-C PUT/GET/decrypt round-trip. nf-driver-deploy gates nf-driver-base on ./gradlew test, which now cascades to :materialiser:test (the Floci spec is @Requires(env.FLOCI_ENDPOINT), so it skips in the gate unless wired).
Where the driver env comes from (the one open wiring). The materialiser reads its wiring (DATAHUB_URL/RUN_ID/BUCKET, the object-store endpoint, NXF_OFFLINE) from the driver Job's env. For a hand-written [Y] ConfigMap template that env is inline; for an [S]-registered template it is the platform [O] half, sourced from config.publishing.env/mounts — which is currently unpopulated, so a registered pipeline expands to an env-less driver until it is filled. That is the last wiring step before the registered → materialise → stock-nextflow path runs end to end; tracked in advaita-declarative-pipeline-publishing-data-model "implementation-status".