What happens inside the driver pod between "the pod starts" and "the bytes land and are recorded", written as a ledger: what each stage consumes, what it produces, which channel carries it, who owns each datum, and the payload shapes in full. The third of the trilogy — advaita-nextflow-router-data-model is the wisp-side ledger (how a submit becomes a Job), advaita-datahub-storage-data-model is the DataHub-side contract (what Allocate/Commit register), and this page is the driver-internal ledger that runs between them. It reconciles to both at its edges: its first stage is the router page's Stage E, its Allocate/Commit calls are the storage page's S3/S4.
The test harness that exercises this model uses three stand-ins, and the labels below name both: DataHub is sim-datahub, CoreWeave object storage is Floci (wisp/k8s/lab/floci.yaml), and the trace receiver is foxfire configured to fan its callbacks at sim-datahub's inbox. The driver, the plugins, and the object bytes are real.
Two authoritative records, joined on run_id:
drive_file rows record what a file is called, where its bytes are, and who may read it.The driver writes into the first and hands a pointer to the second. DataHub's row keeps the hash and a trace_ref back into foxfire; the full signed provenance is a verify call away. Neither side holds a second copy of the other's truth — DataHub does not re-sign the trace, foxfire does not own the file rug.
| Fact | Rug | How the other side sees it |
|---|---|---|
| a run's input→output lineage, each hash | foxfire, signed | DataHub holds metadata.sha256 + trace_ref; expands via GET /runs/:run_id/verify |
| a file's bytes, name, key, ACL | DataHub | the driver holds them transiently, inside one run |
| a produced object's integrity | computed by the driver, witnessed by the bucket, signed into foxfire, recorded by DataHub | four touch points, one value (see D4–D5) |
Inherited from the router and storage pages; the test stand-in is named in parentheses.
| Tag | Owner | Owns |
|---|---|---|
| [C] | client — DataHub as submitter | the request: job_type, params, hub:// refs, tenancy |
| [DH] | DataHub (sim-datahub) | object keys, per-file SSE-C keys, file_ids, the run folder, the ACL decision |
| [CW] | CoreWeave (Floci) | temporary S3 credentials; the bucket's verified checksum — its word on what landed |
| [D] | the driver (nf-advaita-datahub factory + observer, forked nf-amazon) | the bytes, sizes, sha256; the hub://→S3Path translation; signed telemetry |
| [F] | foxfire | the signed trace, the seal chain, verify |
| [I] | platform identity | the wisp-minted run_id, the projected OIDC tokens |
| [Y] | template | the workflow contract |
Three cooperating components, each Nextflow-native, each landing on one DataHub endpoint:
nf-advaita-datahub
├─ FileSystemPathFactory parseUri("hub://…") → resolve → returns an nf-amazon S3Path
│ reads → /inputs/allocate writes → /files/allocate
├─ TraceObserver onFilePublish → captures {object_key, size, sha256} → signed trace + /commit
└─ forked nf-amazon GET/PUT with per-object SSE-C headers, keyed from the resolve stash
The connective tissue is a per-run in-JVM stash: the factory fills it at resolve time (object_key → {file_id, sse_c_key}), the forked nf-amazon reads keys from it at transfer time, and the observer reads file_ids from it at commit time.
| Channel | Plane | Carries |
|---|---|---|
FileHelper.asPath → factory parseUri |
in-process | the hub:// string → an S3Path |
| control plane: HTTPS to DataHub | driver → DataHub | Allocate (read + write), Commit; bearer = datahub-audience token |
| data plane: S3 to CoreWeave | driver → CW | nf-amazon GET/PUT, temp creds + SSE-C headers |
| signed trace: HTTPS to foxfire | driver → foxfire | the signed lifecycle + storage events (key nf-foxfire) |
| callback fan-out | foxfire → subscribers | flattened (unsigned) progress; raw (signed) on request |
The control plane and the data plane never touch, and each is enforced by its own validator: a datahub-audience token presented to CoreWeave fails audience exchange; CW credentials presented to DataHub fail TokenReview.
D0 plugin load ─▶ D1 resolve-in ─▶ D2 stage-in ─▶ TASK ─▶ D3 resolve-out ─▶ D4 put ─▶ D5 record ─▶ D6 telemetry
[factory reg] [/inputs/ [nf-amazon (per output file, ×N) ──────────────┘
allocate] GET+SSE-C] D3 /files/allocate
D4 nf-amazon PUT+SSE-C+checksum
D5 signed trace + /commit
D6 nf-foxfire → callbacks
consumes: the driver image's bundled plugins + nextflow.config
produces: a registered hub:// FileSystemPathFactory + the foxfire TraceObserver
transport: pf4j plugin load at Nextflow startup
The driver Job starts with the datahub-audience and cw-audience tokens projected as files [I], FOXFIRE_CALLBACK_URLS set [C], and aws.client pointed at the object store (Floci in test; the LOTA endpoint deployed). nf-advaita-datahub registers its FileSystemPathFactory and TraceObserver; nf-foxfire registers its signing observer. No control-plane call happens yet.
hub:// → S3Path)consumes: a hub://<file_id> input ref [C]; the datahub token [I]; the run's tenancy [C]
produces: a resolved S3Path + the file's SSE-C key stashed [DH]
transport: params-file → params.sample → Channel.fromPath → FileHelper.asPath → factory.parseUri
The workflow's whole involvement is one scheme-blind line:
ch = Channel.fromPath(params.sample) // params.sample == "hub://df-8f3a…" [C] an opaque file_id
The factory holds the token [I] and the tenancy [C]; it does not hold the object key or the SSE-C key — those are DataHub's [DH]. The ref carries no path or org, so the factory echoes it straight to DataHub, which resolves the file_id and runs the ACL:
# → DataHub POST /runs/{run_id}/inputs/allocate bearer: datahub token
{ input_ref: "hub://df-8f3a…" } # [C] the file_id the user selected, opaque
# ← [DH]
{ object_key: "acme/proteomics/u-31f2/df-8f3a…/<hashed>", # [DH] the SOURCE object
sse_algorithm: AES256,
sse_c_key_b64: "<the file's unwrapped key, this call only>" } # [DH]
DataHub [DH] runs the ACL check at this moment — the run's recorded tenancy [C] against the source file's own tenancy — and only then returns the key. The factory stashes object_key → sse_c_key, and returns S3Path("s3://<bucket>/acme/proteomics/u-31f2/df-8f3a…/<hashed>"). This is the moment the opaque hub:// name becomes a concrete bucket coordinate; the binding file_id df-8f3a… [C] → source object_key acme/…/df-8f3a… [DH] is the origin fact the backward trace depends on.
consumes: the resolved S3Path; the stashed SSE-C key [DH]; ambient CW creds [CW]
produces: a plaintext local file on the work PVC [D]
transport: Nextflow FilePorter → forked nf-amazon GET
nf-amazon GET s3://bucket/acme/proteomics/u-31f2/df-8f3a…/<hashed> [D]
+ SSE-C headers (key from stash) [DH] + ambient CW creds [CW]
CoreWeave/Floci [CW] verifies the key hash → returns plaintext bytes → local file (plaintext boundary)
The observer emits, signed into the trace [D→F]:
:events:input:staged
run_id: img-2026… # [I]
input_ref: "hub://df-8f3a…" # [C] opaque file_id
source_object_key: "acme/proteomics/u-31f2/df-8f3a…/<hashed>" # [DH]
bytes: 1048576 # [D]
The trace now names, signed, where the input came from.
The task runs on the local plaintext file and produces outputs in its task dir (x_fastqc.html, x_fastqc.zip). [D] owns these bytes; no other party has seen them.
consumes: a hub:// publish target [Y]; the datahub token [I]; a produced file's name [D]
produces: a fresh file_id + object_key + per-file SSE-C key, stashed [DH]
transport: publishDir target → FileHelper.asPath → factory.parseUri (per output file)
# → DataHub POST /runs/{run_id}/files/allocate bearer: datahub token
{ name: "x_fastqc.html", contentType: "text/html" } # [D] friendly name
# ← [DH]
{ file_id: "df-01J…", # [DH]
object_key: "acme/proteomics/u-31f2/df-01J…/<hashed>", # [DH] a fresh object; friendly name never in the key
sse_algorithm: AES256,
sse_c_key_b64: "<fresh per-file key>" } # [DH]
The factory stashes object_key → {file_id, sse_c_key} and returns the S3Path.
consumes: the produced file [D]; the stashed key [DH]; temp creds [CW]
produces: the encrypted object in CW, its checksum witnessed [CW]
transport: forked nf-amazon PUT
# nf-amazon PUT s3://bucket/acme/…/df-01J…/<hashed> [D bytes]
x-amz-server-side-encryption-customer-algorithm: AES256 # [DH] key
x-amz-server-side-encryption-customer-key: <sse_c_key_b64>
x-amz-server-side-encryption-customer-key-MD5: <md5>
x-amz-checksum-sha256: <sha256_b64> # [D] — the bucket verifies this inline;
# a mismatched body FAILS at PUT
The driver [D] asserts the hash; the bucket [CW] verifies it against the actual bytes before any record exists, and retains the verified checksum (GetObjectAttributes). Assertion becomes witness.
consumes: the produced file's {object_key, file_id, size, sha256}, the input lineage
produces: a signed trace event [F]; a committed drive_file [DH]
transport: signed trace to foxfire + control-plane Commit to DataHub
The observer's onFilePublish gleans the metadata off the completed transfer and writes it, signed, into the trace [D→F]:
:events:file:written
run_id: img-2026… # [I]
file_id: "df-01J…" # [DH]
object_key: "acme/proteomics/u-31f2/df-01J…/<hashed>" # [DH]
size: 48211 # [D]
sha256: "9c1e…" # [D]
from_input: "hub://df-8f3a…" # [C] the source file_id, stamped onto the output event
from_input on the written event is what makes the origin a single lookup: the output's own trace entry names the input file_id it came from. Then the driver commits to DataHub:
# → DataHub POST /runs/{run_id}/files/{file_id}/commit
{ size: 48211, sha256: "9c1e…",
trace_ref: { run_id: "img-2026…", seq: 14 } } # [D→F pointer] the extra metadata block
# ← { ok: true }
DataHub [DH] records drive_file { …, status: READY, metadata: { run_id, sha256, trace_ref } } and may cross-check sha256 against [CW]'s witness for the cost of a metadata call. Four touch points now agree on one value: the producer [D] computed it, the bucket [CW] verified it, foxfire [F] signed it, DataHub [DH] recorded it. DataHub keeps the hash and the pointer; the full signed provenance is GET foxfire /runs/{run_id}/verify.
Build note. The sha256 the observer records is available two ways: re-hash the local source file (the observer holds source on onFilePublish), or read what the forked client computed for the x-amz-checksum-sha256 header and stashed. Re-hashing the source is self-contained and needs no stash coordination for the value; the stash still carries the file_id.
consumes: the run's lifecycle + task + storage events [D]
produces: progress + completion at each subscribed callback
transport: nf-foxfire callback fan-out (FOXFIRE_CALLBACK_URLS)
nf-foxfire fans each event to the callback URLs it was handed at launch, shaping each to the consumer's registered projection: flattened (unsigned) for a status view, raw (the signed envelope) for a consumer that wants the guarantee. In the test, foxfire points at sim-datahub, whose inbox folds the flattened stream into run status. The receiver only receives — the emission and the routing are the driver/foxfire side. A "facts always reach DataHub regardless of who launched the run" rule, if wanted, is a foxfire-side fan-out configuration or a re-fanning receiver, never the simulator reaching outward.
From a final recorded object, the source is one chain, every hop owner-labeled:
recorded object s3://bucket/acme/proteomics/u-31f2/df-01J…/<hashed> [CW holds it]
│ object_key → DataHub drive_file df-01J… [DH]
│ drive_file.metadata.{ run_id, trace_ref } [DH → pointer]
▼
foxfire trace run_id (GET /runs/{run_id}/verify → seal chain) [F, signed]
│ :events:file:written { file_id df-01J…, sha256, from_input: "hub://df-8f3a…" }
│ :events:input:staged { input_ref "hub://df-8f3a…" → source_object_key acme/…/df-8f3a… }
▼
source object s3://bucket/acme/proteomics/u-31f2/df-8f3a…/<hashed> [CW holds it]
│ object_key → DataHub drive_file df-8f3a… [DH]
▼
the original file the user selected (Phase 0), file_id df-8f3a… [C]
The input ref is opaque — hub://<file_id> carries no path, so the whole chain is joined on file_ids and run_id, not on human-readable names. run_id is the spine, foxfire's signed trace is the ledger that names input→output lineage, the object keys are the coordinates, and DataHub's rows anchor ownership at each end. When that chain reads end to end — output object → run_id → trace → input file_id → source object → the selected file — the model is correct.
Before any of the above, the user selects files in DataHub and DataHub asks wisp what can run on them. Today the answer is wisp's template catalog projected up: GET /templates serves each template's params schema + data_inputs, and DataHub renders it as an offer, confirms a selection, and submits with parameters (the storage page's S0). The R0 offer engine (drive-service PR #30) is the interim shape of this on the DataHub side. This phase contributes only which template and which file refs downstream, both of which reappear labeled at D1, so a mock may serve a fixed template list without changing anything below.
The choreography is testable before the encryption is:
x-amz-server-side-encryption-customer-* headers + the checksum header, keyed per object from the stash. Floci's SSE-C enforcement proves the encrypted path, including the negative case.