A straw-man for a shared, multi-app pipeline-execution architecture: the components, how they communicate, and what each is responsible for. Boxes are roles, not final ownership; dashed edges and the closing Open forks list mark what's still to decide, and each fork's working preference is called out inline as a Lean.
flowchart LR
subgraph Apps["Consumer apps"]
DH[DataHub]
ISG[iScanGuide]
end
L["Shared Launcher<br/>queue • capacity policy<br/>stamps origin (org/user/app)"]
subgraph K8s["Kubernetes cluster"]
DJ["Driver Job (ephemeral)<br/>Nextflow pipeline"]
end
FR["foxfire receiver<br/>cluster-wide event sink"]
MET(["Metering store<br/>job + cost records"])
BILL["Billing / rating<br/>(consumes the meter)"]
OBJ[("CoreWeave Object Storage<br/>SSE-C encrypted")]
KMS[["AWS KMS<br/>master CMK"]]
DH -- "launch request<br/>(pipeline, params, origin)" --> L
ISG -- "launch request" --> L
L -- "render + fire Job" --> DJ
DJ -- "read inputs / write outputs" --> OBJ
DJ -- "unwrap data key (compute-read)" --> KMS
DJ -- "signed run/task/cost events" --> FR
DJ -- "terminal callback" --> L
L -- "status / completion" --> DH
L -- "status / completion" --> ISG
FR -. "usage + cost feed" .-> MET
L -. "job lifecycle + timing" .-> MET
MET -- "attributable meter" --> BILL
datahub-launcher (wanderland) vs. adopt
Milan's sc-task-queue vs. hybrid.The Driver Job box above is a single unit to the Launcher, but at runtime — with the
k8s executor (the norm) — it expands. The driver pod runs Nextflow as the workflow
scheduler and submits one k8s pod per pipeline task, each carrying resource requests
declared by the pipeline (nf-core process_* labels → conf/base.config). The k8s
scheduler places each task pod by those requests; the task pods share the per-run scratch
volume and hand intermediates through it.
flowchart TB
L["Shared Launcher"] -->|"fire ONE Driver Job<br/>+ create per-run namespace + ResourceQuota (budget)"| RUN
subgraph RUN["Per-run namespace — isolation (D7) · quota (D3) · metering boundary (D5)"]
DJ["Driver pod<br/>(Nextflow head = workflow scheduler)"]
DJ -->|"submit a pod per task<br/>(requests from process_* labels)"| T1["task: bclconvert<br/>(process_high)"]
DJ --> T2["task: fastp<br/>(process_medium)"]
DJ --> T3["task: Falco / MultiQC<br/>(process_low)"]
T1 -. share .-> SCR[("per-run scratch PVC<br/>RWX · block-encrypted")]
T2 -. share .-> SCR
T3 -. share .-> SCR
end
KS["k8s scheduler"] -.->|"places each task pod<br/>by declared requests"| RUN
RUN -->|"per-pod cost events (actuals)"| FR["foxfire → meter"]
RUN -->|"terminal callback"| L
Who knows the fan-out's resource needs — three schedulers, only one of them ours:
queueSize / per-process maxForks.Convergence: the per-run namespace is the natural unit for three dimensions at once — scratch isolation (D7), the fan-out budget/quota (D3), and the metering boundary (D5) (usage summed across the run's task pods by namespace).
Metering refinement (D5): with fan-out, resource accounting must aggregate all task pods, not just the driver — the per-run namespace makes that a clean sum.
Single-pod fallback: with the local executor there is no fan-out — the same Driver Job runs every step inside the driver pod against a pod-scoped ephemeral volume (strongest D7 isolation). It's the fallback for small or especially-sensitive runs; the fan-out above is the norm.
| # | Dimension | What it means |
|---|---|---|
| D1 | Origin identity | assign + propagate org / user / app / pipeline+version / run_id |
| D2 | Queue & dispatch | admission, concurrency caps, cross-app arbitration (fair-share / priority) |
| D3 | Capacity & allocation | node pools, SPOT vs reserved, batching, per-app quotas |
| D4 | Execution lifecycle | render/fire Jobs, retries, timeouts, preemption, status |
| D5 | Cost metering | timing (queue/exec) + resources (CPU/GPU-sec, mem, node class) + outcome |
| D6 | Observability / events | signed run/task/cost events, tracing |
| D7 | Data security | SSE-C at rest, compute-read key unwrap, encryption context, compute-read scratch isolation |
| D8 | Callback / results | terminal callback + results delivered back to the app |
| D9 | Billing / rating | apply a rate card to the meter → charges / chargeback |
| D10 | Driver image & workflow packaging | how the workflow attaches to the base driver image: runtime-mounted ("live") vs. baked, version-pinned sub-images |
| Component | D1 | D2 | D3 | D4 | D5 | D6 | D7 | D8 | D9 | D10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Consumer apps (DataHub, ISG) | C | · | · | · | · | · | · | O | · | · |
| Shared Launcher | O | O | O | O | C | · | · | C | · | C |
| Driver Job (ephemeral) | C | · | · | C | C | O | C | C | · | C |
| foxfire receiver | · | · | · | · | C | O | · | · | · | · |
| Metering store | · | · | · | · | O | · | · | · | C | · |
| Object storage (CoreWeave) | · | · | · | · | · | · | O | · | · | · |
| KMS (master CMK) | · | · | · | · | · | · | O | · | · | · |
| Driver image / workflow build (platform) | · | · | · | · | · | · | · | · | · | O |
| Billing / rating (future) | · | · | · | · | · | · | · | · | O | · |
NF_DRIVER_IMAGE and picks the workflow), and the Driver
runs it — both contribute, but the packaging strategy is a platform/build-time
decision. Ties to the repo-home fork (extend foxfire/driver/ vs. a dedicated
advaita-nf library).Since cost is the new emphasis, this is where the responsibility question is sharpest. Each meter field, and its natural source:
| Meter field | Source component |
|---|---|
| org / user / app / pipeline+version / run_id | Launcher stamps; Driver echoes into events |
| enqueued → started (queue time) | Launcher |
| started → finished (execution time) | Launcher / Driver |
| CPU-seconds, GPU-seconds (+ GPU type) | Driver → foxfire cost events (pod usage) |
| memory GB-seconds | Driver → foxfire cost events |
| node class (Reserved vs SPOT), node pool | Launcher (placement decision) + k8s node labels |
| status, retries, preemptions | Launcher / Driver |
Takeaway: the meter is assembled from two feeds — the Launcher (timing +
placement) and foxfire (resource usage). Whoever owns the Metering store
joins them. Milan's sc-task-queue already holds per-job state, so it's the
natural host — with foxfire as the usage feed.
A base image (Nextflow + nf-k8s + core driver config) is common to all runs;
the fork is how the workflow attaches:
| Option | How | Pro | Con |
|---|---|---|---|
| Runtime-mount ("live") | workflow via ConfigMap/volume into the base image | fast iteration, no rebuild, one image | weak provenance — workflow not pinned to an image digest |
| Baked sub-images | per-pipeline image FROM base, workflow COPY'd, version-tagged |
immutable provenance (digest pins tool + workflow + reference versions) | more images to build/manage, slower iteration |
Lean: both, split by environment — runtime-mount for dev (speed), baked + version-pinned for prod (scientific reproducibility, esp. Cell Ranger + reference data). This dimension also feeds cost/provenance: the baked image digest is what makes a billed run reproducible after the fact.
Compute can't run on ciphertext — the pipeline must decrypt inputs to a local scratch volume to process them, so every run has a window where plaintext customer data sits on disk. And since fan-out (k8s executor) is the norm — a driver pod plus per-step task pods sharing a work dir — that scratch is a shared (RWX) volume, not one pod's private disk. Isolating it is a first-class part of D7, not an afterthought:
run_id and delete it on the terminal callback — never a long-lived PVC shared
across runs/tenants. It's shared only among that run's pods, then wiped. (Needs an
orphan reaper for runs that never call back.)emptyDir.medium: Memory) for stages that fit in RAM;
Nextflow cleanup where -resume isn't needed.Inherent limit: some pod always holds plaintext during compute — you can't run bcl-convert on ciphertext — so the goal is bounding blast radius to one run, ephemeral, encrypted, destroyed on completion, not eliminating plaintext.
Lean: a per-run, block-encrypted scratch volume whose lifecycle is bound to the run and torn down on the terminal callback. (A single-pod local-executor run gets the strongest isolation — a pod-scoped ephemeral volume, no sharing at all, auto-wiped with the pod — and stays the fallback for especially sensitive or small runs that don't need fan-out.)
Ownership: the Launcher provisions + tears down the per-run volume (it already owns lifecycle D4 and fires the Job); the Driver runs on it; the platform supplies the volume-encryption + storage class.
datahub-launcher vs. adopt sc-task-queue vs. hybrid.