The storage round-trip between DataHub and the compute plane, written as a ledger: what each stage consumes, what it produces, which channel carries it, and the payload shapes in full. Companion to advaita-datahub-identity-and-storage-design (the argument and the security reasoning) and advaita-nextflow-router-data-model (the compute-plane ledger these stages wrap around).
The design was approved at review 2026-07-17, and every numbered decision is now closed: #1 key/path granularity — per-file (one SSE-C key per file, shareable across the parts of that file's multipart upload; DataHub's existing data model, kept intact); #2 temp-cred scope — per-org runner SA + bucket policy (S2); #3 commit granularity — per-file (S4); #5 input — allocate-read (S3). #4 was a confirmation rather than a decision, and both halves are confirmed (S4) — what remains of it is the build itself, tracked in the ledger.
Two sources of truth, two domains, joined on run_id:
drive_file rows, sidecars, and wrapped keys are the authoritative record of what exists, what it's called, and who may read it.Every view that spans both — the inbox that marks a job SUCCEEDED, the run folder listing its outputs — is a projection that reconciles to one of the two rugs, never a third copy (lebowski-corollary). The seam is explicit:
| Fact | Rug | How the other side sees it |
|---|---|---|
| run status, progress, cost | foxfire | inbox consumes the signed events (a derived view) |
| a file's bytes, name, key | DataHub | the driver holds them only transiently, inside one run |
| a file's integrity (sha256) | DataHub, recorded at Commit | computed by the driver, verified by the bucket, recorded by DataHub — see File integrity under S5. The trace cannot hold it: foxfire's hash field is Nextflow's task cache key (inputs + script + container), not a content hash of outputs |
| run ↔ file lineage | both, joined on run_id |
drive_file.sidecar.provenance.run_id ↔ foxfire's run record |
Inherited from the router page: [C] client · [I] platform identity · [F] foxfire history · [P] profiler · [Y] template · [K] Kueue. Three storage actors join them:
| Tag | Source | Mints |
|---|---|---|
| [DH] | DataHub | object keys, per-file/run SSE-C keys, file_ids, the run folder, job correlation |
| [CW] | CoreWeave | temporary S3 credentials; the bucket's word on what landed |
| [D] | the driver | bytes, sizes, sha256s, signed telemetry (key nf-foxfire) |
The storage flow adds no new wisp channel. It uses the five channels the router page names, plus two planes of its own:
| Channel | Plane | Carries |
|---|---|---|
POST /submit body |
wisp (existing) | DataHub is a client [C]: job_type, params (knobs), data (the input map), callback_url = the inbox |
template identity_token + Job volumes |
wisp (existing) | the projected OIDC tokens — the credential channels |
| overlay ConfigMap | wisp (existing, prospective here) | per-run aws.client {} (LOTA endpoint, virtual addressing) + app-scope config for the nf plugin |
| control plane: HTTPS to DataHub | new, driver → DataHub | Allocate / Commit, bearer = datahub-audience token |
| data plane: S3 to CoreWeave | new, driver → CW | nf-amazon PUT/GET, temp creds + SSE-C headers |
| signed events + callback fan-out | foxfire (existing) | completion + progress into the inbox |
The two planes never touch, and the separation is enforced by each side's own validator rather than by convention. A datahub-audience token presented to CoreWeave fails signature exchange — the WIF trust is configured for the https://coreweave.com/iam audience, and audience mismatch rejects the token before any policy is consulted. CW credentials presented to DataHub fail TokenReview — they are S3 signing material, not a cluster-issued JWT, and the Allocate/Commit endpoints accept nothing else. A compromise of either credential therefore yields one plane's capability, never both: the datahub-audience token can register metadata for its own run but move no bytes; the CW credentials can move bytes but register nothing.
S0 submit ─▶ S1 launch ─▶ S2 creds ─▶ S3 input ─▶ run ─▶ S4 output ×N ─▶ S5 records
[DH→C] [Y+I] [I→CW] [DH]+[CW] [DH]+[CW]+[D] [DH]
└────────▶ S6 telemetry ─▶ inbox [D→DH]
DataHub is a wisp client like any launcher. The submit body carries two distinct collections, and the distinction matters downstream: params are knobs — values the workflow reads (thresholds, study names, modes) — and data is the input map — named keys holding URIs for files the workflow consumes. Params are validated against the template's declared schema; data keys are validated against the template's declared data_inputs. Both are opaque to the router's routing logic; both are checked at the door.
consumes: user action ("run pipeline on this file"), drive_file refs [DH]
produces: the submit record [C]; DataHub-side run correlation [DH]
transport: POST /submit (wisp, existing)
# → wisp POST /submit
app_id: datahub
job_type: img.recontrast_run # [C] a registered template
immediacy: deferred # [C]
tenancy: # [C] who this run belongs to — asserted by DataHub,
org_id: acme # the authenticated policy decision point; field
team_id: proteomics # names match the drive_file schema (org_id NOT
owner_id: u-31f2 # NULL = the tenant fence; team_id nullable — the
# file's pinned team; owner_id = the user). org_id
# selects the runner SA (see Tenancy); the whole
# block lands on the run record
params: # [C] knobs — validated against the template schema
study: ST000405
threshold: 0.05
data: # [C] the input map — named keys → URIs, any scheme
sample: "hub://acme/proteomics/x.mzML" # [DH] a DataHub file → the path factory
ref_db: "s3://public-refs/uniprot/2026.fasta" # plain S3 → stock nf-amazon
callback_url: "http://datahub.…/api/pipeline/inbox" # [C] → FOXFIRE_CALLBACK_URLS
# ← wisp response
run_id: img-20260718093012-4c1f2e # [I] the join key for everything below
Both collections travel to the workflow as one document: wisp writes params.json (validated params, defaults applied, plus the data map) as a second key in the per-run overlay ConfigMap it already ships, and the driver argv gains -params-file /etc/wisp/params.json. Nextflow populates params.* from the file identically to --flag argv, so workflow code is indifferent to the transport. The template consequently declares what the job accepts rather than how values travel: a params schema (a YAML-authored JSON Schema subset — types, required, enum, ranges, defaults, descriptions; the nf-core nextflow_schema.json pattern) and a data_inputs block naming each expected input. Validation runs at submit: a missing required input or out-of-range param is a 422 before Kueue admits anything, and GET /templates serves the schemas as the discovery catalog — "what can you do" is the template list, "how do I do that" is the schema. (Build note: wisp currently hydrates params through per-template ERB argv holes; the params-file + schema form replaces that — see the build ledger.)
# DataHub-side, same transaction [DH]
pipeline_run:
run_id: img-20260718093012-4c1f2e
job_id: job-… # DataHub's own job row
status: SUBMITTED
dest_folder: folder-… # ensureFolder(run outputs) — created up front
Everything in the router page's Stage E holds; the storage flow adds the credential mounts. The template declares them; classify renders them.
consumes: template identity_token declaration [Y]
produces: driver Job with two projected OIDC tokens + the callback env
transport: Job manifest (volumes + env), existing channels
# template delta [Y]
driver:
identity_token: # BUILD NOTE: today this block mounts ONE
- audience: datahub # token; the two-token form needs the small
mount_path: /var/run/secrets/datahub # list extension in JobManifest
- audience: "https://coreweave.com/iam"
mount_path: /var/run/secrets/coreweave
# rendered onto the driver pod
volumes:
- projected serviceAccountToken { audience: datahub, path: token } # → Allocate/Commit
- projected serviceAccountToken { audience: "https://coreweave.com/iam",
path: token } # → S2, read by the AWS SDK
env:
- FOXFIRE_CALLBACK_URLS: "http://datahub.…/api/pipeline/inbox" # from S0 [C]
- AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE: /var/run/secrets/coreweave/token
- AWS_CONTAINER_CREDENTIALS_FULL_URI: "https://api.coreweave.com/v1/cwobject/temporary-credentials/oidc/{orgId}"
Both tokens are kubelet-rotated files bound to the driver pod. Neither crosses to the other system: audience scoping is the wall. (CoreWeave also ships a Pod Identity Webhook that injects the volume + env pair at admission — the option for task pods, should they ever need direct S3; with the PVC work dir, S3 transfers run in the driver, so the template block covers today's topology.)
Verified against CoreWeave docs (2026-07-17). CKS clusters are OIDC identity providers, and AI Object Storage accepts OIDC Workload Identity Federation: the exchange endpoint is shaped to match the AWS container-credentials provider, so the AWS SDK's default credential chain performs the exchange — and the re-exchange on expiry — by itself. There is no credential client to build and nothing for the nf plugin to refresh.
consumes: projected SA token, aud https://coreweave.com/iam [I]
(via AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE)
produces: temporary S3 credentials [CW], auto-refreshed by the AWS SDK
transport: GET https://api.coreweave.com/v1/cwobject/temporary-credentials/oidc/{orgId}
(raw JWT as Authorization; driven by the SDK, not by our code)
# what the SDK receives and manages [CW]
accessKeyID: CWTEMP…
secretKey: …
expiry: 2026-07-18T10:30:12Z # SDK re-exchanges before expiry; no plugin code
The identity is structural, not a bearer secret: policies name the principal as role/[OIDC-ISSUER]:system:serviceaccount:<namespace>:<sa>, verified against the cluster's own issuer. Nothing rides through params, argv, env values, or logs — the token is a file the kubelet rotates.
OPEN #2 — resolved: the scoping model is known. CoreWeave evaluates organization access policies first, then bucket policies (set via the S3 API). cwobject:* actions must use * as the resource; s3:* actions scope to buckets and key prefixes, and bucket-policy Condition blocks with an explicit Deny give real prefix isolation (Deny beats any later Allow).
# the recommended posture — per-ORG service account, audit-based run isolation
principal: system:serviceaccount:vivarta-drivers:<org-runner-sa> # stable, per org
org boundary: IAM — bucket policy scopes the org SA to the org's bucket/prefix
run isolation: NOT IAM — declared output prefix (the plan) vs observed writes,
reconciled through foxfire; run-level write pollution is a
correctness bug inside one org's trust boundary, caught by audit
fence intact: cross-run READ impossible (no SSE-C key) · cross-run REGISTER
impossible (Commit is run-bound)
Per-run IAM (a bucket-policy Deny per run prefix) is expressible but churny — a policy write per run for a boundary the ledgers already witness. The wisp seam for the per-org principal exists: service_account is already a per-template/env argument.
Three cheap verifications before this is closed (the spike list):
nf-amazon defers to the SDK default credential chain (a test pod running aws s3api list-buckets with only the env pair proves the exchange; a trivial s3:// Nextflow run proves nf-amazon inherits it). 3. The can-i endpoint (POST /v1/cwobject/auth/can-i) as the reconciliation probe for the offer engine's storage half.consumes: the data map [C] — named keys → URIs (hub:// s3:// https:// ftp://);
datahub token [I]; ambient CW creds [CW]
produces: each input staged into the run's work dir [D]
transport: params-file → params.<key> → file()/fromPath → scheme-dispatched
filesystem providers; hub:// adds one control-plane call per input
The workflow consumes each input by name and treats it as a file, never as a location:
// the workflow's entire involvement with input plumbing
ch_sample = Channel.fromPath(params.sample) // hub://, s3://, ftp:// — same line
process ANALYZE {
input: path sample // staged by Nextflow; a local file inside the task
...
}
Three separate mechanisms cooperate here, and none of them is ours to build except the first:
Scheme dispatch. Nextflow maps each URI scheme to a filesystem provider. s3:// is stock nf-amazon (riding S2's ambient credentials), http(s):// and ftp:// are built-in read-only providers, and hub:// is the path factory: its parseUri makes the resolve call below and returns an S3Path, after which the transfer is stock nf-amazon again. One plugin, one scheme; every other scheme is upstream code.
Resolution — URI plus key, never URI rewriting. A hub:// reference cannot be pre-translated to s3:// at submit time, because resolution returns two things and only one of them is an address:
# → DataHub POST /api/pipeline/{run_id}/input:allocate bearer: datahub token
{ input_ref: "hub://acme/proteomics/x.mzML" }
# ← [DH]
{ object_key: "acme/proteomics/u-31f2/df-8f3a21…/<hashed_file_name>", # the source object
sse_algorithm: AES256,
sse_c_key_b64: "…" } # the file's unwrapped SSE-C key, this call only
# then: nf-amazon GET s3://bucket/u/…/8f3a21… with SSE-C headers + ambient creds [CW]
The object is SSE-C encrypted, so a bare s3:// URI is unreadable; the key must travel with the resolution. Resolving at runtime, inside the driver, is what makes the auth work (the run-bound datahub token exists only there), keeps the key just-in-time (never in the submit body, argv, or params-file), and keeps DataHub's bucket layout private. OPEN #5 is resolved this way in the approved design — allocate-read, symmetric with output. Presigned GETs remain the right primitive for the browser path into DataHub; the compute plane does not use them.
Staging. Where a fetched file lands is Nextflow's decision (the FilePorter downloads foreign files into the work dir's staging area and links them into task directories), made per task, per retry, per resume. No layer of ours promises landing paths, and the workflow developer never sees one — task cache keys are computed from source URI and content, so pre-computed paths would buy nothing and break -resume the first time conventions diverged.
Inside the run, staged inputs and produced outputs exist as plaintext on the work PVC — Nextflow scratch space. The approved design accepts this explicitly, with two boundary conditions. At rest in object storage, everything is SSE-C (CoreWeave offers no SSE-KMS; customer keys are the only server-side option, which is why DataHub minting and wrapping them is structural, not stylistic). And the work volume must not outlive its trust boundary: a volume that serves runs from different users or orgs carries run A's plaintext into run B's mount. Today's topology is a single shared RWX nextflow-work volume across all runs — plus the shared nextflow-cache volume holding session DBs — which is fine for the lab and a real gap for multi-org tenancy. The isolation options (per-org volumes with org-scoped namespaces, or per-run volumes with a lineage-preserving cache home) are a platform build item, tracked in the ledger below; the always-shared cache design already anticipated the launch-dir topology changing under it.
The heart of the flow. For each file the workflow publishes, the nf plugin's path factory runs the three-step dance. Bytes go driver → CoreWeave direct; DataHub sees two small JSON calls.
consumes: a produced file [D]; datahub token; temp creds [CW]; SSE-C key [DH]
produces: the encrypted object in CW [CW]; a committed drive_file [DH]
transport: control plane ×2 + data plane ×1, per file
# 1 — allocate
# → DataHub POST /api/pipeline/{run_id}/files:allocate bearer: datahub token
{ name: "recontrast_summary.tsv", # [D] friendly name — recorded at Commit, never in the S3 key
contentType: "text/tab-separated-values" }
# ← [DH]
{ file_id: "df-01J…", # the eventual drive_file identity
object_key: "acme/proteomics/u-31f2/df-01J…/<hashed_file_name>", # the drive layout:
# <orgId>/<teamId>/<ownerId>/<fileId> prefix (V11) +
# hashed name — tenancy-prefixed, friendly-name-free,
# and org-first, so the per-org bucket policy scopes
# on the leading path segment for free
sse_algorithm: AES256,
sse_c_key_b64: "…" } # minted + KMS-wrapped DataHub-side; raw here only
# 2 — put (data plane; no DataHub involvement)
# nf-amazon PUT s3://bucket/runs/…/01J… with SDK-managed creds [CW] and:
# aws.client (overlay): endpoint http://cwlota.com (in-cluster LOTA proxy),
# s3PathStyleAccess: false (virtual addressing: <bucket>.cwlota.com)
x-amz-server-side-encryption-customer-algorithm: AES256
x-amz-server-side-encryption-customer-key: <sse_c_key_b64>
x-amz-server-side-encryption-customer-key-MD5: <md5>
x-amz-checksum-sha256: <sha256_b64> # [D] the bucket verifies this in-line —
# a mismatched upload FAILS at PUT (see S5)
# 3 — commit
# → DataHub POST /api/pipeline/{run_id}/files/{file_id}:commit
{ size: 48211, sha256: "9c1e…" } # [D] the same hash, now for the record
# ← { ok: true } # DataHub may cross-check it (see S5)
Key + path granularity — resolved: per-file (#1, closed 2026-07-17). Each output file gets its own SSE-C key, minted and KMS-wrapped at Allocate — DataHub's existing data model, kept intact. One key may span the parts of a single file's multipart upload (SSE-C requires the same customer key on every UploadPart and the CompleteMultipartUpload of that object), but a key never spans files. Object keys stay opaque, one per file; the friendly name is recorded at Commit and never appears in the S3 key.
allocate: once per output file — mints file_id + opaque object_key + fresh SSE-C key
key scope: one file (all parts of that file's multipart upload share the key)
key blast radius: one file
friendly name: recorded at Commit → drive_file.name; never in the S3 key
Commit granularity — drafted per-file (#3): the approved interface contract commits each file as it lands (files/{file_id}:commit), which streams naturally and leaves partial results registered if a run dies at file N of M. The batch-manifest alternative (one atomic registration at run end) remains available if atomicity of the output set turns out to matter:
# as drafted — per-file, as each file lands
POST files/{file_id}:commit { size, sha256 }
# the alternative if set-atomicity is required — one manifest at run end
POST /api/pipeline/{run_id}/files:commit
{ files: [ { file_id: df-01J…, size: 48211, sha256: "9c1e…" },
{ file_id: df-01K…, size: 1204, sha256: "77af…" } ] }
The fork's exact scope (#4 — closed as a question; open only as work). The store side is settled: CoreWeave AI Object Storage supports SSE-C with the standard S3 headers — AES-256 for encryption, SHA-256 for key verification, and the key itself is never stored (only its hash is retained, to reject a wrong key at GET). Any S3 client that can send the three x-amz-server-side-encryption- customer-* headers works. ("Never stored" is server-side: the key rides each request's headers and lives in the client's memory only for the duration of the transfer; CoreWeave keeps the key's hash, not the key, so it can reject a wrong key at GET without ever being able to decrypt on its own.) The plugin side defines the fork: stock nf-amazon exposes aws.client.storageEncryption with exactly two values, AES256 (SSE-S3) and aws:kms — there is no per-object customer-key path in the released plugin. The fork's delta is therefore precise and small: thread the three SSE-C headers (and the checksum header above) through the SDK client for PUT, GET, and the multipart operations, keyed per object from what Allocate returned. Nothing else in the transfer machinery changes.
consumes: the allocate row + the commit payload
produces: drive_file + sidecar in the run folder [DH]
transport: DataHub-internal (reuses UploadService's mint + commit path)
drive_file: # [DH] — the file rug (files.drive_file, V4 + V8 + V11)
id: df-01J… # uuid pk
org_id: acme # NOT NULL — the tenant fence
team_id: proteomics # nullable — the file's pinned team (V11)
owner_id: u-31f2 # NOT NULL — the user
folder_uri: folder-… # the run's dest folder from S0
original_file_name: recontrast_summary.tsv # friendly name, from Allocate/Commit
hashed_file_name: 9f2c… # what actually appears in the object key
prefix: acme/proteomics/u-31f2/df-01J… # <orgId>/<teamId>/<ownerId>/<fileId>
size_bytes: 48211 # [D] via Commit
content_type: text/tab-separated-values
status: READY
encrypted: true # per-object marker (V8) — downloads send SSE-C headers
wrapped_data_key: <b64> # the SSE-C data key, wrapped under the ORG master
key_ref: acme-master-v3 # wrapping-key id/version, for rotation
metadata: { run_id: img-20260718093012-4c1f2e, job_type: img.recontrast_run,
sha256: "9c1e…" } # jsonb — run lineage + integrity (see note below)
sidecar: # index.json — ONE per prefix, deliberately unencrypted
recovery: # lets DR recover the blob without the DB row:
wrapped_key: <b64> # the wrapped data key
org_id: acme # the KMS encryption context
key_ref: acme-master-v3
Two schema notes from the code review (develop @ 2026-07-17). The wrapping is per-org, not per-env: ObjectCrypto wraps each data key under the org's master with org_id as the KMS encryption context — Drive is the sole KMS caller, which is the "tenant master" the key-management standard's encryption half describes. And the Commit integrity value lands in the existing metadata jsonb, not a new column (decision 2026-07-17): a content hash reads as metadata, and jsonb lets the receiver stamp sha256 — plus anything the interface grows later — without a migration per field. drive_file has no sha256 column today and gets none; the receiver writes it into metadata alongside the run lineage.
Download is DataHub's existing path: unwrap the key via KMS, stream the object with SSE-C, decrypt to the caller. The driver is long gone; nothing it held survives the run.
The sha256 on the drive_file row is the file's integrity record: the fixed point that every later question — is this download intact, did replication preserve the bytes, has anything rotted at rest — reconciles to. Three parties touch it, in order, and each plays a distinct role:
The driver computes it. The nf plugin hashes the bytes as it streams them to CoreWeave in step 2. The value appears twice on the wire: once in the x-amz-checksum-sha256 header on the PUT, once in the Commit payload. The driver is the only party that ever sees the plaintext bytes and the hash together, so it is necessarily where the hash originates — but a value that originates with the writer is an assertion, not a verification.
The bucket witnesses it. CoreWeave AI Object Storage supports the same checksum algorithms as AWS S3, SHA-256 among them, verified on upload: a PUT whose body does not hash to the declared x-amz-checksum-sha256 value fails at the bucket, before any record exists anywhere. The verified checksum is stored with the object and is retrievable afterwards through GetObjectAttributes or HeadObject — both supported. This turns the driver's assertion into something the object store has independently checked against the actual bytes it holds.
DataHub records it. Commit delivers {size, sha256} over the run-bound control plane. Because the bucket has already verified the same value, DataHub can cross-check the assertion against the witness for the cost of a metadata call — GetObjectAttributes on the just-committed key, compare the stored checksum to the Commit payload — with no byte transfer at any size. A match means three parties agree: the process that produced the bytes, the store that holds them, and the registry that names them. A mismatch fails the Commit, which is the correct failure: a file whose producer and store disagree about its content must not become a drive_file.
The threat model each layer addresses: the checksum-verified PUT catches corruption in transit (a flipped bit between driver and bucket fails the upload, retried by the SDK); the recorded hash catches corruption at rest and in later transit (any download can be re-hashed against the record); the cross-check at Commit catches a driver that computes or reports its hash wrongly — a buggy hasher, a race between write and hash, or a compromised image. What remains outside the model is a driver that produces wrong but internally consistent output — garbage bytes, correctly hashed. No integrity mechanism addresses that; it is what workflow testing, the signed trace, and run reproducibility (resume_of + the recorded session lineage) exist for.
This is the plan-vs-placement pattern applied to bytes: the driver's books (the asserted hash), the store's witness (the verified checksum), and a recorded reconciliation of the two — no party trusted alone, disagreement surfaced at the moment of record rather than discovered at read time.
consumes: the run's lifecycle + task events [D] (flattened projection)
produces: pipeline_job status transitions + live progress [DH]
transport: callback fan-out — FOXFIRE_CALLBACK_URLS, driver-direct, low-latency
The driver fans every event to the inbox as it fans the same events to foxfire — verified live on the worked-example run, where echo-backend's /callbacks receives the identical stream. The fan-out shapes each event to the consumer's registered projection: the foxfire receiver takes raw (the verbatim signed envelope, which it verifies and folds); the inbox registers flattened — the normalized, unsigned record — because its trust boundary is the deploy-level fence (ingress path-deny + the vivarta-drivers NetworkPolicy) and its authoritative source is foxfire itself (GET /runs/:run_id), not an edge signature check.
The raw envelope and the flattened record the inbox consumes:
# raw — the signed envelope foxfire ingests; `payload` is a JSON STRING (nested,
# camelCase), and there is no top-level run_id or seq. Callbacks may opt into raw.
key_id: nf-foxfire
to: ":events:run:echo-20260717144738-d94b94"
from: ":events:run:echo-20260717144738-d94b94"
type: ":events:run:started"
payload: "{\"runId\":\"echo-…\",\"runName\":\"echo-…\",\"sessionId\":\"fee93b5e-…\",\"event\":\"started\",\"utcTime\":\"2026-07-17T14:47:52Z\",\"metadata\":{…}}"
trace_hash: "c73503e9959ef2…" # SHA-256 of the payload string
signature: "MEQCID6nhJhMWs…" # EC-P256 over key_id\nto\nfrom\ntype\ntrace_hash
# flattened — what the inbox receives: payload hoisted to the top level,
# snake_case, unsigned (no payload/trace_hash/signature).
type: ":events:run:started"
event: started
run_id: echo-20260717144738-d94b94 # = payload.runName (the wisp run_id, what the inbox stores)
run_name: echo-20260717144738-d94b94
session_id: fee93b5e-3fd3-45e5-bb84-e09e5d436b6b
utc_time: "2026-07-17T14:47:52Z"
# run:completed adds top-level: exit_status (int), success (bool), duration_ms
There is no seq on the wire, raw or flattened: foxfire assigns a per-run seq when it appends an event to its store (event_store.rb: the count of prior events), so seq lives on foxfire's stored records and never travels on the callback. The inbox correlates on run_id and orders on the events themselves.
# inbox state machine [DH]
:events:run:started → pipeline_job.status = RUNNING; record session_id
:events:task:submit → n_submitted += 1
:events:task:start → n_running += 1
:events:task:complete → n_running = max(0, n_running - 1); n_done += 1
:events:task:cached → n_done += 1 # cached tasks skip submit/start
:events:run:completed → status = SUCCEEDED | FAILED (top-level exit_status);
outputs already registered by S4/S5 commits
The inbox is a derived view of foxfire's rug. Status correctness rides on run:started and run:completed plus a terminal guard — a terminal job drops every later event, so an at-most-once fan-out (the driver ships each event once, with no retry) can neither resurrect a finished run nor reorder its ends. The counters are best-effort live progress: a dropped task event skews only them. n_running is a gauge (started minus completed), and a cached task counts toward n_done without ever running. Recovery is never local: on a gap, a stall, or an inbox that missed events, the inbox reads foxfire's run record (GET /runs/:run_id, the same read the resume path uses) and adopts what the rug says — verified against the seal chain (GET /runs/:run_id/verify) when it wants the signed guarantee the flattened push does not carry. The callback is best-effort delivery of a stream whose complete, signed copy always exists at foxfire, so a flaky network degrades the inbox's latency, never its correctness; the pathological cases (inbox down for an hour, DataHub redeployed mid-run) all resolve to the same recovery: read the rug.
A future hardening opts the inbox into raw and enforces the EC-P256 signature at the edge as defense-in-depth (the verifier seam already takes the raw bytes). That is not this milestone; the fence plus foxfire reconciliation is the trust model today.
Every driver → DataHub call is validated the same way:
consumes: bearer token (projected SA token, aud datahub)
produces: { authentic, sa_identity, pod_name } via in-cluster TokenReview
guards: ROLE_DRIVER (the SA) · pod_name startsWith(run_id) (run binding)
Why the pod name is available to check. A projected ServiceAccount token is a bound token (Kubernetes KEP-1205): the kubelet requests it for a specific pod, and the token embeds that binding as claims. When DataHub presents the token to TokenReview, the API server verifies the signature against the cluster's own keys, confirms the bound pod still exists, and expands the binding into status.user.extra:
# TokenReview response (the fields the auth filter reads)
status:
authenticated: true
user:
username: system:serviceaccount:vivarta-drivers:nextflow-runner
extra:
authentication.kubernetes.io/pod-name: [echo-20260717144738-d94b94-vvzqh]
authentication.kubernetes.io/pod-uid: [061469b0-…]
authentication.kubernetes.io/node-name: [gf2a9c4]
authentication.kubernetes.io/credential-id: [JTI=…]
Two properties follow. First, the binding is live, not merely signed: a token whose pod has been deleted fails review even inside its TTL — a stolen token dies with its pod. Second, the pod name is asserted by the API server from the binding, not parsed out of the JWT by DataHub, so the run check is a string comparison against an authenticated field: extra["authentication.kubernetes.io/pod-name"][0].startsWith(run_id).
Why startsWith(run_id) is sound. wisp names the Job exactly run_id (router page, Stage B), and Kubernetes derives Job pod names as <job-name>-<suffix>. Job names are DNS-1123 labels bounded well under the 63-character limit by the run_id format (<lane>-<14-digit stamp>-<6 hex>), so the prefix is never truncated. A run-A driver therefore cannot Allocate or Commit against run B: its authenticated pod name carries run A's prefix.
Remaining caveats, precisely stated. The pod-claim expansion is standard for pod-bound tokens on current Kubernetes; the pre-sync spike is a five-minute on-cluster confirmation that the CKS API server populates the authentication.kubernetes.io/* extra keys (one TokenReview against a test pod's projected token), not a design risk. The token proves which pod, not which image — a driver pod runs whatever image its Job specified; the mitigation is that only wisp holds Job-create in vivarta-drivers, and the reaper's observed image_id digest (placement ledger) provides the audit trail. The stronger future bind is a run-scoped audience (datahub:<run_id>), minted per-Job through wisp's existing audience-parameterized identity_token block, which moves the run binding from a naming convention into the token's own audience claim.
FastQC is the concrete first target: an existing spike (nf-fastqc-driver + Drive's pipeline package + datahub-launcher) already runs the whole loop, so the integration is a migration of a working thing onto this model, not a greenfield build. What exists today, and what each piece becomes:
The workflow is already model-clean. workflows/fastqc/main.nf reads params.input, writes params.outdir, declares one FASTQC process with publishDir — it is indifferent to how params arrive, so the -params-file + data map transport (S0) drops in with no workflow change. The data map here is one input (sample) and the config's aws.client gains the LOTA endpoint + virtual addressing.
Two transfer paths exist, and the choice is the first decision. The plain driver (bin/datahub-run) lets nf-amazon do S3 through Channel.fromPath/publishDir — the target shape, pending the SSE-C fork (#4). The SSE-C variant (bin/datahub-run-ssec) sidesteps the fork entirely: it does aws s3api get-object/put-object with --sse-customer-key in the bash wrapper, downloading to the work dir before Nextflow runs and uploading after. That means the encrypted first integration can ship before the nf-amazon fork lands — the wrapper is the bridge — and the fork becomes the follow-up that moves SSE-C inside nf-amazon so publishDir works transparently. PipelineKeyService already returns null (keyless launch) when encryption is off, so an unencrypted first run needs neither the fork nor the wrapper's key path.
The control-plane calls replace the spike's shortcuts, one for one:
| FastQC spike (today) | This model | Why |
|---|---|---|
launcher POSTs --input/--outdir as s3:// on the Job |
wisp /submit with data map; wisp resolves + launches |
one launch surface, tenancy-checked, sized/queued |
plaintext SSE-C keys in the launch POST body (LauncherClient) |
keys never leave DataHub; driver gets them JIT from input:allocate per file |
keys off the launch path entirely |
| per-run output key (V13), stamped on every output row | per-file key at files:allocate (decision #1) |
matches DataHub's per-file model; columns (V8) already exist |
datahub-run shell POSTs a terminal callback with outputKeys[] + CALLBACK_TOKEN |
per-file Commit (run-bound token) + signed foxfire events to the inbox |
integrity at commit; progress + completion on the signed stream |
poller claims jobs with locked_by/locked_at; correlates on run_id |
wisp launches directly; run_id still the correlation key |
removes the poll/claim loop; run_id semantics unchanged |
FastqcOutputIngestService fences output keys to <prefix>/fastqc/ |
same fence, now at Commit against the run's recorded tenancy | keeps the trailing-slash-guarded prefix check; adds tenancy |
What next week's integration actually needs, smallest first: the data map + params-file in wisp (unblocks a keyless FastQC run end to end); the tenancy block + org→SA resolution; the Allocate/Commit receiver reshaped from the held presigned receiver; pipeline_job.team_id (V14) and the metadata.sha256 write. The SSE-C fork and per-org bucket-policy hardening follow — the wrapper and keyless launch carry the first run without them.
A run belongs to an org, a team, and a user, and the question "where does the token learn that" has a precise answer: it mostly doesn't, on purpose. The granularities are enforced at different layers, and only the coarsest one lives in the OIDC identity.
Verified against the code (advaita-drive-service, develop, 2026-07-17). The tenancy triple exists in the schema exactly as this section needs it: drive_file.org_id (NOT NULL — the migration comments call it the tenant fence), drive_file.owner_id (NOT NULL — the user), and drive_file.team_id (nullable — the file's pinned team, added in V11 with the prefix column). The object-key layout is tenancy-first: <orgId>/<teamId>/<ownerId>/<fileId> plus the hashed file name, which means the per-org CW bucket policy scopes on the leading path segment with no layout change. Org membership resolves through an operator-managed email-domain → org_id mapping (V6, read-through cached). One fix scheduled for the integration: the current pipeline_job table records org_id + owner_id but not team_id (V12). The run record must carry all three, so pipeline_job gains team_id (sourced from the input file's pinned team_id, or the submitting user's active team) and Allocate's ACL checks read it directly rather than re-deriving team per call.
The authorization decision happens before submit, in DataHub. DataHub authenticates the user, checks its own ACLs — may this user run this job type, may they read these input files — and only then submits. wisp never re-derives user permissions; it receives a run whose inputs were already authorized by the system that owns the file rug. The submit body carries the result of that decision as the tenancy block: an assertion by DataHub (the authenticated caller), recorded onto the run, not proven by any token the driver holds.
Org granularity is the OIDC token's job — and its only job. The structural principal is system:serviceaccount:<namespace>:<sa>, so the identity can carry exactly as much tenancy as the SA name encodes. The per-org posture (S2) means org onboarding mints a runner-<org> ServiceAccount and a CW bucket policy granting it that org's bucket/prefix — once per org, never per run. At Job render, wisp resolves tenancy.org → serviceAccountName from its org registry; the pod starts as that SA; the kubelet mints tokens naming it; CoreWeave's pre-provisioned policy does the rest. The caller never names an SA (a caller-supplied SA would be privilege escalation — wisp derives it), and nothing is issued at runtime: selecting the ServiceAccount is selecting the credential.
Team and user granularity never reach CoreWeave — they are enforced by Allocate. This is the piece that looks missing until you see where it lives. Even with org-wide CW read credentials, every object at rest is SSE-C encrypted: the bytes are unreadable without the per-file key, and the only source of per-file keys is DataHub's run-bound input:allocate. So the SSE-C key is the per-file capability, and DataHub checks each grant at the moment of allocation: this run's recorded owner (the tenancy block from submit) against the file's ACL. A user who may read a team file gets its key; the same driver asking for another team's file gets a 403 from Allocate — and its CW credentials alone can only fetch ciphertext. Fine-grained authorization stays in the system that owns the file rug, evaluated per file, per run, at request time.
There is no "here are your credentials" step. The sequence, end to end: DataHub authorizes the user and submits with tenancy → wisp resolves the org's runner SA and renders the Job → the pod starts and the kubelet materializes its identity as rotating token files → the SDK exchanges the CW-audience token against pre-provisioned org policy (S2) → the driver asks Allocate per file, and each answer is a capability scoped to one object. The run_id is a correlation key and a naming bind (the pod-name check), never a bearer of authority; nothing is populated after submit, because identity was fixed at render and capabilities are dispensed per file.
One honest limit: within an org, two teams' runs share the runner SA's CW write scope, so cross-team write pollution (not reads — keys gate those) is possible at the object layer and is caught by the plan-vs-observed audit rather than IAM, same as cross-run pollution (S2). If a tenant ever needs harder intra-org isolation, the same machinery extends: runner-<org>-<team> SAs and prefix policies, at the cost of a bigger registry — the mechanism doesn't change, only the granularity of the onboarding.
| Piece | Owner | Depends on | Status |
|---|---|---|---|
submit/poller swap → wisp /submit |
DataHub | nothing (decision-invariant) | buildable now |
| telemetry inbox (S6) | DataHub | nothing (decision-invariant) | buildable now |
| Allocate/Commit receiver (S4/S5) | DataHub | nothing (all decisions closed) | reshaped from the held run-scoped presigned-multipart receiver → Allocate + Commit; reuses UploadService's key-mint + row/sidecar internals, drops the presign step; T4-SECURITY's run-bound token still gates it. Replaces the FastQC-spike transport (plaintext input/output SSE-C keys in the launch POST body, per-job callback_token bearer, poller with locked_by claims — LauncherClient/PipelineJobPoller) and moves output keys from the spike's per-run mint (pipeline_job.output_wrapped_data_key, V13, stamped identically on every output row) to the per-file mint of decision #1 — the per-file columns (wrapped_data_key/key_ref on each row, V8) already exist. Commit's sha256 lands in the existing metadata jsonb — no new column |
| nf path factory (+ nf-amazon SSE-C fork) | platform | nothing (confirmed both sides) | collapses: with ambient SDK creds it is a FileSystemPathFactory — resolve → S3Path, no FileSystemProvider, no signed URLs, no multipart code |
two-token identity_token (list form) + AWS_CONTAINER_* env |
platform (wisp) | nothing | small JobManifest extension |
| — | — | deleted: the AWS SDK container-credentials provider does the exchange + refresh (S2, verified) | |
| per-org runner SA + bucket policy | platform + DataHub | resolved posture (S2) | wisp service_account arg already per-template/env |
params-file + data map + template schemas |
platform (wisp) | nothing | replaces per-template ERB run_opts with invariant argv + -params-file in the existing per-run ConfigMap; templates gain params schema + data_inputs; submit-time 422 validation; GET /templates becomes the discovery catalog (S0) |
| driver/base images gain the storage plugins | platform | fork + factory built | the nf-driver family base image bundles the path factory + forked nf-amazon; every driver image rebuild picks them up |
| image build cadence (Jenkins) | platform | nothing | drivers auto-build on commit; the base image rebuilds on a schedule (~weekly) to pick up Nextflow fixes; float-vs-pin-minor versioning decision deferred until the cadence exists |
tenancy in the submit contract + org→SA resolution |
platform (wisp) + DataHub | org registry | submit carries {org, team, user}; wisp resolves org → runner SA at Job render (callers never name an SA); the tenancy block lands on the run record for Allocate's ACL checks |
| org onboarding (SA + bucket policy) | platform + DataHub | nothing | once per org: mint runner-<org> SA + CW bucket policy for the org's bucket/prefix; the registry wisp resolves against |
pipeline_job.team_id migration |
DataHub | nothing | V14: add team_id (source: input file's pinned team); the run record carries the full tenancy triple for Allocate's ACL checks |
| work-volume tenancy | platform | multi-org rollout | shared nextflow-work/nextflow-cache volumes must not span org trust boundaries (S3, plaintext boundary); per-org or per-run volumes with a lineage-preserving cache home |
| WIF enablement + nf-amazon chain smoke test | platform | nothing | the S2 spike list |
| TokenReview pod-name claim-key check | platform | nothing | five-minute on-cluster confirmation (auth spine) |