advaita-datahub-identity-and-storage-design# DataHub ↔ nf driver — consensus design (summary)

Date: 2026-07-17 · Diagram-backed summary of 2026-07-17-datahub-driver-consensus-design.md for the sync. Draft to converge on — the Open decisions at the bottom are what we still need to agree.

The shape of it

The driver writes pipeline outputs directly to CoreWeave object storage — no bytes through DataHub, no DataHub-minted presigned URLs. It gets temporary S3 credentials from its own projected OIDC token (CoreWeave mints them), and uploads with nf-amazon (a light fork). DataHub keeps everything DataHub-specific by exposing two small control-plane calls the driver makes around each file: Allocate (DataHub returns the object key + a per-file SSE-C key it mints and KMS-wraps) and Commit (DataHub finalizes the drive_file + sidecar). Completion and progress ride nf-foxfire's signed telemetry, fanned to a DataHub inbox. The nf-side glue is a path factory + s3-driver fork, not a full datahub:// filesystem. Net: bytes go direct (scales to GB outputs), DataHub still owns layout, keys, records, and download.

The round-trip

sequenceDiagram
    autonumber
    participant U as User / DataHub UI
    participant DH as DataHub
    participant W as wisp
    participant DRV as nf driver Job
    participant CW as CoreWeave Object Storage
    participant FOX as foxfire

    U->>DH: run pipeline on this file
    Note over DH: ensureFolder(dest), mint run correlation
    DH->>W: POST /submit {job_type, immediacy, params, callback_url=DH inbox}
    W-->>DH: {run_id}
    W->>DRV: launch driver Job — mount 2 OIDC tokens (aud coreweave.com/iam, aud datahub), FOXFIRE_*, callback

    rect rgb(232,238,248)
    Note over DRV,CW: CREDS — driver mints temp S3 creds from its own OIDC token
    DRV->>CW: exchange OIDC token (aud coreweave.com/iam) for temporary S3 creds
    CW-->>DRV: temp access key + secret + session token
    end

    rect rgb(198,219,247)
    Note over DRV,CW: INPUT — allocate at DataHub, read direct from CoreWeave
    DRV->>DH: POST input:allocate (aud datahub, run-bound) name+ref
    DH-->>DRV: object_key + per-file SSE-C key
    DRV->>CW: GET object (SSE-C) via nf-amazon + temp creds
    end

    DRV->>DRV: nextflow run (k8s executor to task pods)

    rect rgb(200,236,206)
    Note over DRV,CW: OUTPUT per file — allocate, write direct, commit
    DRV->>DH: POST files:allocate (aud datahub) name
    DH-->>DRV: object_key opaque + per-file SSE-C key (DataHub keeps the KMS-wrapped form)
    DRV->>CW: PUT object (SSE-C) via nf-amazon + temp creds — direct, no DataHub bytes
    DRV->>DH: POST files/{id}:commit size + sha256
    Note over DH: finalize drive_file + sidecar in the run folder
    end

    par Telemetry — signed lifecycle events
        DRV->>FOX: submit / start / complete (EC-P256 signed)
    and Progress + completion to DataHub via callback fan-out
        DRV->>DH: FOXFIRE_CALLBACK_URLS same signed events
    end
    Note over DH: run completed event → mark job SUCCEEDED
    DH-->>U: outputs in the run folder, live progress from the event stream

Who owns what

flowchart LR
    subgraph DHside[DataHub owns]
      SUB[submit client to wisp]
      ALLOC[Allocate and Commit endpoints]
      KMS[per-file SSE-C key mint plus KMS wrap]
      REG[drive_file plus sidecar plus download-decrypt]
      INBOX[telemetry inbox for completion and progress]
      OFFER[offer engine and generic form]
    end
    subgraph PLAT[Platform owns]
      WISP[wisp classify launch Kueue]
      PLUGIN[nf plugin path factory plus nf-amazon SSE-C fork]
      DRIVER[driver image plus workflows plus 2 OIDC tokens]
      FOXP[nf-foxfire signed telemetry]
    end
    CW[(CoreWeave Object Storage SSE-C at rest)]
    FOXR[(foxfire receiver)]

    SUB -->|POST /submit| WISP
    WISP -->|launch Job| DRIVER
    DRIVER --> PLUGIN
    PLUGIN -->|allocate and commit aud datahub| ALLOC
    PLUGIN -->|temp creds via OIDC aud coreweave| CW
    PLUGIN -->|PUT and GET SSE-C direct| CW
    ALLOC --> KMS
    ALLOC --> REG
    FOXP -->|signed events| FOXR
    FOXP -->|callback fan-out| INBOX

Interface contract (draft)

#	Endpoint (DataHub)	Auth	Request → Response
1	POST /api/pipeline/{run_id}/files:allocate	datahub-aud token, run-bound	{name, contentType} → {file_id, object_key, sse_algorithm, sse_c_key_b64}
2	(write → CoreWeave directly)	CW temp creds (OIDC)	nf-amazon PUT + SSE-C
3	POST /api/pipeline/{run_id}/files/{file_id}:commit	datahub-aud token, run-bound	{size, sha256} → {ok}
4	POST /api/pipeline/{run_id}/input:allocate	datahub-aud token, run-bound	{input_ref} → {object_key, sse_c_key_b64}
5	POST /submit (exists) · foxfire telemetry inbox	— · signed events	as already specified

Auth: the driver mounts two projected OIDC tokens — aud: coreweave.com/iam (→ CW temp creds, bytes) and aud: datahub (→ Allocate/Commit, run-bound via the T4-SECURITY pod-name binding). Neither token crosses to the other system.

Security guarantees — how DataHub trusts a driver call

Every driver→DataHub call carries the driver pod's projected Kubernetes ServiceAccount token (aud: datahub), which DataHub validates in-cluster via TokenReview. That gives four guarantees:

• Authentic — the token is cluster-signed and checked against the cluster key by TokenReview; it can't be forged.
• Scoped to DataHub — the datahub audience is enforced, so a coreweave.com/iam or default-audience token can't be replayed here.
• Is a driver — the validated identity is the driver SA; /api/pipeline/** requires ROLE_DRIVER.
• Is this run's driver — the token's bound pod-name (surfaced by TokenReview) must startsWith(run_id); since wisp names the Job = run_id, a run-A driver can't act on run B. (A run-scoped audience datahub:<run_id> is the stronger future bind.)

No shared secret, nothing DataHub-minted, short-lived + auto-rotated; the cluster is the trust anchor. Two caveats: it proves "a pod named <run_id>-* running as the driver SA is calling," not which image that pod runs (supply-chain — mitigated by only wisp being able to create driver Jobs); and pod-name binding must be confirmed to appear in the cluster's TokenReview or drivers fail closed. CoreWeave byte access is a separate plane — the driver's coreweave.com/iam token → CW temp S3 creds, never touching DataHub.

Open decisions — need consensus at the sync

• Key + path granularity. Recommend per-run prefix + per-run SSE-C key, opaque object names, friendly name recorded at Commit — one Allocate + one key (the platform's minimalism), names stay out of the S3 key, DataHub model intact; blast radius of the shared key is just that one user's run.
• CW temp-cred scope — the biggest security item. Temp creds from a shared runner-SA OIDC identity likely grant broad object-store write (a driver could write under other runs' prefixes = pollution) — but cannot read others' encrypted data (no SSE-C key) or register cross-tenant (Commit is run-bound). Confirm whether CW IAM can scope creds to the run prefix; else accept the pollution risk (Allocate/Commit + per-run keys are the real fence) or keep DataHub-presigned writes as the prod fallback.
• Commit granularity — per-file (simple, streams) vs a batch manifest at run end.
• nf-amazon SSE-C — confirm the fork can set the per-object customer key (AWS SDK supports it).
• Input — Allocate-read + nf-amazon (symmetric) vs a DataHub presigned-GET.

Where this leaves the build

• Decision-invariant, buildable now (no dependence on #1–#2): the submit/poller swap to wisp /submit, and the foxfire telemetry inbox (completion + progress).
• Waits on #1–#2: the Allocate/Commit receiver (reuses UploadService's key-mint + row/sidecar commit; drops the presign step). T4-SECURITY's run-bound datahub-audience token still gates it.

Research resolution — CW temp-cred scope (2026-07-17)

Open decision #2 is resolved in mechanism, verified against CoreWeave docs. AI Object Storage supports OIDC Workload Identity Federation (CKS clusters are OIDC IdPs, Nov 2025); the exchange endpoint GET /v1/cwobject/temporary-credentials/oidc/{orgId} matches the AWS container-credentials provider shape, so a projected SA token (aud https://coreweave.com/iam) plus the AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE / AWS_CONTAINER_CREDENTIALS_FULL_URI env pair gives the driver SDK-managed, auto-refreshing S3 credentials — no exchange or refresh code anywhere. The principal is structural (system:serviceaccount:<ns>:<sa> via the cluster issuer), so nothing DataHub-minted rides the workflow layer. Scoping: organization access policies evaluate first, bucket policies (S3 API) second; s3:* actions scope to buckets and key prefixes with Condition + explicit Deny. Recommended posture: per-org runner SA as the IAM boundary; run-level write isolation via plan-vs-observed reconciliation through foxfire, not per-run IAM. In-cluster transfers should use the LOTA endpoint (http://cwlota.com, virtual addressing). Remaining spikes: org WIF enablement, nf-amazon → SDK default-chain smoke test, TokenReview pod-name. Full contracts: advaita-datahub-storage-data-model §S2.

Status — approved at review 2026-07-17

The design was approved at review. Remaining open: #1 key/path granularity (steer: per-run acceptable) and #4 confirm the nf-amazon SSE-C fork. Resolved: #2 temp-cred scope (per-org runner SA + bucket policy; ambient SDK-managed creds per the research resolution above), #5 input (allocate-read, symmetric with output); #3 is drafted per-file in the interface contract. The submit input model was extended after review: a top-level data map (named keys → URIs) beside params, carried via -params-file in the per-run ConfigMap, with template-declared schemas (params + data_inputs) validated at submit — contracts in advaita-datahub-storage-data-model §S0/§S3.

#1 closed — per-file (2026-07-17, post-review read-through). One SSE-C key per file, minted at Allocate; a key may be shared across the parts of that file's multipart upload but never across files. This is DataHub's existing data model, kept intact. With #4 confirmed on both sides (CW SSE-C support; stock nf-amazon has no customer-key path, so the fork threads the headers), no numbered decisions remain open — everything left is build work, tracked in advaita-datahub-storage-data-model's ledger.