The PKB governed build written as a ledger: what each Job consumes, what it produces, which channel carries it, and the record it folds into the signed trace. Companion to advaita-nextflow-router-data-model (the launch this build uses), advaita-datahub-storage-data-model (the storage sibling, same two-rugs shape), and pkb-coreweave-compute-plane (the design home and the trace-envelope reasoning). Model of record for the ADV-455 prototype; tracks the handoff contract from the PKB dev (case task-0e952514, note 2026-07-20).
PKB is one immutable multi-platform image with four fixed Job interfaces: acquire (plan / resolve / fetch), seal, build, and promote. Nextflow orchestrates those interfaces, captures the JSON results and immutable artifacts each writes, and folds their provenance into the signed run. The command graph inside each Job belongs to the image.
So the workflow selects a profile_id and passes governed documents — an acquisition-spec, a build-template, a release_id — and the image validates them and owns their contents. The acquisition profile_id must equal the build-template's profile_id; a profile / source / handler / receipt mismatch fails before any acquisition work. profile_id is an opaque value the image validates, not an enum the workflow encodes: two profiles exist today (pkb-production-full-v1, 29 sources; pkb-production-ppi-v1, 14 sources / 12 steps), and more are intended, so the workflow treats the id as a string and lets the image own the closure.
Nextflow selects profile_id · passes acquisition-spec + build-template + release_id + run_id
│
▼
JA acquire plan → resolve → fetch
│ network: controlled egress · /incoming ro (governed PVCs) · netrc ro
│ writes → /acquisition (one private attempt, root-locked, atomic publish)
│ records → per-source receipt (origin, bytes, sha, row counts, receipt sha)
▼
JB seal offline · no egress · no secrets · no /incoming
│ rehash the exact profile closure
│ writes → /acquisition/releases/<release_id> (atomic, read-only thereafter)
│ records → acquisition-release-manifest, pkb-build-spec.json, terminal attempt, receipts
▼
JC build offline governed · /inputs ro (sealed release) · /var/lib/pkb rw · /scratch/pkb rw
│ preflight → plan → build
│ writes → immutable candidate (separate from the current pointer)
│ records → candidate DuckDB hash, package manifests, package-gate, PPI stats TSV, completion manifest
▼
JD promote explicit, authorized action after gates + human approval
records → promotion status, current pointer before / after
Every Job uses the same immutable image digest. The acquire Job is the only one with network; seal and build carry explicit no-egress NetworkPolicies. Each Job is a Nextflow task, so each gets its own execution crossing (nf-foxfire) and can emit its own domain crossings — all folding on run_id.
Inherited from the router page: [C] client / caller · [I] platform identity (run_id) · [Y] template · [K] Kueue · [F] foxfire (recorded history). Three actors specific to this build join them:
| Tag | Source | Mints |
|---|---|---|
| [PK] | the PKB image | the governed content — resolved release metadata, per-source receipts, the sealed build-spec, the candidate DuckDB, package manifests, the package-gate report, the completion manifest, and every hash over them |
| [NF] | nf-foxfire | the execution envelope — per-task submit / start / complete, metrics (peak_rss, realtime, exit), signed and folded automatically |
| [D] | the driver's emit | the fold of a [PK] document into a signed foxfire crossing on this run_id — the passthrough that carries an authored fact into the trace (never a derived or detected one) |
[PK] and [NF] are distinct on purpose, the same way [F] and [P] are on the router page. nf-foxfire records what executed — it owns that, because it scheduled the pods. The PKB image records what the governed content is — it owns that, because it computed the bytes and the hashes. The driver's job at the seam is to fold the image's authored documents into the run, not to infer them.
Two sources of truth, joined on run_id + release_id (lebowski-corollary):
| Fact | Rug | How the trace sees it |
|---|---|---|
| task timing, RSS, exit | foxfire [NF] | the execution envelope, emitted per task |
| a source's bytes + integrity | the image [PK] | the driver folds the per-source receipt [D]; the sha binds the artifact |
| the sealed release contents | the image [PK] | the acquisition-release-manifest sha + build-spec sha, folded [D] |
| the candidate package | the image [PK] | package-manifest hashes + gate status + stats-TSV hash, folded [D] |
| run ↔ release ↔ candidate lineage | both | run_id on the crossings ↔ release_id in the sealed manifest ↔ candidate pointer |
The build adds no new wisp channel. It uses the channels the router page names, with values shaped for a governed run.
| Channel | Written by | Carries |
|---|---|---|
POST /submit body |
caller [C] | job_type (pkb.build / pkb.promote), params (profile_id, release_id, data_cutoff_date, spec paths), data (governed source refs), tenancy, callback_url |
params-file (params.json) |
wisp | the governed run identity the workflow reads — profile_id, release_id, run_id, data_cutoff_date, the spec paths |
| overlay ConfigMap | wisp | per-process resources + the queue-name / run-id / process labels |
| volumes + pod securityContext | template [Y] | the mounts and the container contract (below) |
| network posture label | template [Y] | egress: controlled (acquire) or egress: deny (seal, build) — a standing NetworkPolicy / CiliumNetworkPolicy selects it |
| projected identity token | template [Y] | audience-scoped SA token for the governed data sources / object store |
| signed events | nf-foxfire [NF] + the emit [D] | the execution envelope and the folded domain crossings, into foxfire and any callback endpoint |
The mounts are the container contract, declared in the workflow's baked nextflow.config as native pod directives per process:
| Mount | Mode | Jobs | Holds |
|---|---|---|---|
/acquisition |
rw | acquire, seal | the private acquisition attempt, then the sealed releases/<release_id> tree |
/incoming |
ro | acquire | governed local / PVC source lanes (no egress needed) |
/run/secrets/kegg/netrc |
ro | acquire (full profile) | MEDICUS netrc, when the profile requires it |
/inputs |
ro | build | the sealed release, mounted read-only |
/var/lib/pkb |
rw | build, promote | the persistent build volume — candidate + current pointer |
/scratch/pkb |
rw | build | scratch |
Every Job runs 10001:10001, non-root, read-only root filesystem, drop-all capabilities, no-new-privileges. The volumes must be Linux/Kubernetes-compatible storage (no dependence on APFS semantics).
The point of the ledger for a governed build is that every byte has a named origin and a receipt. This is the per-artifact record — what each file is, where it came from at each step, which Job wrote it, where it lives, and what the trace records about it.
| Artifact | Origin | Written by | Lives at | Trace record |
|---|---|---|---|---|
| upstream source download | http / ftp fetch over controlled egress (the network sources) | JA fetch | /acquisition/<source>/ |
per-source receipt [PK→D]: canonical origin URL, retrieval timestamp, compressed bytes + sha256 |
| governed local package | PVC volume, read-only under /incoming (the local sources) |
JA fetch (stage) | /acquisition/<source>/ |
per-source receipt [PK→D]: governed PVC manifest identity, deterministic directory-tree sha256 |
| resolved release metadata | resolve call over controlled egress — floating latest/current upstream surfaces pinned to a concrete release |
JA resolve | /acquisition/… |
resolved acquisition-spec sha256 (so the sealed build never consumes a floating identity) |
| sealed release | rehash of the exact profile closure (offline) | JB seal | /acquisition/releases/<release_id> (ro thereafter) |
acquisition-release-manifest sha, pkb-build-spec.json sha, one receipt sha per profile source |
| candidate DuckDB | image compute, offline, from /inputs |
JC build | /var/lib/pkb (candidate) |
candidate DuckDB hash + bytes; pre- and post-build input attestation |
package (ipg_dpus_t0) |
image compute | JC build | candidate package dir | package-manifest hashes; package-gate report hash + status; PPI stats TSV hash |
| completion manifest | image compute | JC build | candidate | completion-manifest hash |
| promoted pointer | pointer move, authorized | JD promote | /var/lib/pkb current pointer |
promotion status; current pointer before / after |
Two origins, one receipt shape. A network source and a governed-PVC source differ only in the origin field of the receipt — a canonical URL for the fetch, a PVC manifest identity for the mount — and both carry the same integrity fields (bytes, sha256, row counts, receipt sha). The trace reads identically whichever lane a source came from; the lane is a fact recorded in the receipt, not a fork in the reporting.
The PPI output boundary constrains what the candidate may contain: a T0 export only, never T700; original STRING v12 link scores retained as provenance; the IPG package plus its stats and provenance sidecars, and no RDS / cache artifact, no automatic DPUS/AE cache creation, no LIANA/ISG or metabolomics package in a PPI-only candidate. The Rat identity release stays a separate, inert sidecar — it is never mounted as /inputs and never treated as an automatically promotable PPI source.
submit ─▶ JA acquire ─▶ JB seal ─▶ JC build ─▶ JD promote
[C+I] [PK]+[NF] [PK] [PK]+[NF] [PK] (human-gated)
│ │ │ │
└────────────┴───────────┴────────────┴──▶ foxfire (run_id) : envelope [NF] + domain crossings [D]
The workflow provides a profile and governed documents; it does not provide the source list or the command graph. Grant's rule, made a validation contract:
# → wisp POST /submit
job_type: pkb.build # [C]
params: # [C] the governed run identity
profile_id: pkb-production-ppi-v1 # opaque; the image validates it
release_id: rel-ppi-20260720 # names the sealed release
run_id: (minted by wisp) # [I] the fold key
data_cutoff_date: 2026-07-07
acquisition_spec: /inputs/acquisition-spec.json # a governed document path
build_template: /inputs/build-template.json
data: {} # [C] governed source refs, when a source arrives by ref
tenancy: { org_id: advaita, owner_id: … } # [C]
# MUST NOT provide: an arbitrary source-ID list · a subset of build commands ·
# shell/argv in a spec · floating source paths in the sealed spec · auto-promotion ·
# a cache threshold disguised as an export threshold.
The image validates the pair before any writable attempt or network call: the acquisition profile_id must equal the build-template's profile_id, and a profile / source / handler / receipt mismatch fails closed.
consumes: acquisition-spec + build-template [C]; controlled egress; /incoming ro; netrc ro
produces: one private acquisition attempt → a complete staging tree [PK]
transport: image entrypoint /usr/local/bin/pkb-acquire; /acquisition rw
pkb-acquire plan --acquisition-spec … --build-template …
# resolve: pin floating latest/current upstream surfaces to concrete release metadata
pkb-acquire fetch --acquisition-spec … --build-template … \
--acquisition-root /acquisition --incoming-root /incoming
plan asserts the closure (14 sources, 12 steps, only ipg_dpus_t0 for PPI) before work. resolve pins mutable upstream surfaces so the sealed build consumes a concrete identity. fetch performs one private attempt over a root lock and publishes atomically — no partial release is ever published. Each source produces a receipt [PK]; the driver folds each into the run [D] (per-source receipt fields under Signed trace fields). A real PPI acquisition fails closed while its exact handler closure is incomplete — the prototype's WOULD-RUN mode is correct during that interval.
consumes: the acquisition attempt [PK]; no network, no secrets, no /incoming
produces: /acquisition/releases/<release_id>, published atomically [PK]
transport: pkb-acquire seal --acquisition-root /acquisition
Seal rehashes the exact profile closure and adds pkb-build-spec.json, acquisition-release-manifest.json, the terminal attempt record, and one source receipt per profile source. Network is disabled by both container configuration and a Kubernetes NetworkPolicy. The sealed release is the fixed point the build consumes: read-only, hash-bound, floating-free.
consumes: the sealed release at /inputs (ro) [PK]; /var/lib/pkb rw; /scratch/pkb rw
produces: an immutable candidate, separate from the current pointer [PK]
transport: pkb-container {preflight,plan,build} --spec /inputs/pkb-build-spec.json
pkb-container preflight --spec /inputs/pkb-build-spec.json
pkb-container plan --spec /inputs/pkb-build-spec.json
pkb-container build --spec /inputs/pkb-build-spec.json
Build runs the exact trusted command plan the spec names, offline, and emits the candidate DuckDB, the package manifests, the package-gate report, the PPI stats TSV, and the completion manifest [PK] — each hash-bound and folded [D]. The candidate is immutable and distinct from the promoted pointer.
consumes: a candidate that passed its gates + human/release approval [C]
produces: the current pointer moved [PK]
transport: pkb-container promote --spec /inputs/pkb-build-spec.json
Promotion is a separate, authorized submit — a distinct pkb.promote job_type, never automatic after a build. --replace-current is reserved for an explicitly authorized replacement after all completion checks. The trace records promotion status and the current pointer before and after.
Trace splits along the same seam as the two rugs: the layer the driver owns is captured automatically; the layer the image owns is emitted explicitly.
nf-foxfire is a TraceObserverV2 (FoxfireFactory → FoxfireObserver). It hooks Nextflow's lifecycle callbacks — onFlowCreate, onTaskSubmit / Start / Complete / Cached, onFlowComplete — signs each into the four-address envelope, and ships it keyed on run_id (the run name wisp passes via -name; the session id travels as lineage). enableMetrics() is on, so the envelope carries peak_rss, realtime, and exit. This is the execution record, and it is complete without any workflow change: the driver scheduled the pods, so the driver's plugin owns what they did.
The observer's own callbacks are the whole of what TraceObserverV2 exposes — there is no hook the workflow script can call to record a domain fact at an arbitrary point. That capability is the SPI below.
The domain facts (receipts, manifests, hashes, the command plan, profile_id, release_id) are authored by the PKB image as JSON artifacts. Folding them into the run is a passthrough: read the authored document, wrap it in the same signed envelope on the same run_id, ship it. This reuses nf-foxfire's envelope core exactly as FoxfireObserver.emit does — Envelope.contentHash(payload) → Envelope.signable(keyId, to, from, type, traceHash) → Signer.sign → Projection.raw / flattened → EventShipper — under a domain type (:events:pkb:source-receipt, :events:pkb:release-sealed, :events:pkb:build, :events:pkb:promote). Because it signs with the same key_id and folds on the same run_id, foxfire joins it to the execution envelope with no receiver change.
Two implementable forms, both feeding on the image's authored JSON:
foxfireEmit(type:, payload:) that reuses its Signer / Envelope / Projection / EventShipper and the session's run_id / key_id. The workflow calls it at any point — e.g. receipts.map { foxfireEmit(type: ':events:pkb:source-receipt', payload: parseJson(it)) } after a process outputs receipt.json. The developer writes whatever they like, wherever they like, with no extra pod and no driver change when the payload grows a field. This is the driver growing the capability at the layer that already owns signing, shipping, and run identity.foxfire_emit boundary (already the Wanderland→foxfire bridge for CI/CD crossings on the PUBLISH path) into a small task that folds an authored document. Heavier (a task pod per emit) but itself an execution step, which suits coarse artifacts and the CI/CD flow that already runs in a Wanderland pod.The driver never parses PKB's stdout to infer a domain fact. Detection would recreate the image's command graph at the seam — the thing Grant's contract forbids — it would be brittle against output changes, and it could scrape a credential or a licensed row into the trace. Explicit emit signs what the image asserts (the authored receipt), so the signature is meaningful and the "do not put these in the trace" rule is enforced at the one place that chooses the payload. The observer stays the auto execution layer; the emit is the explicit domain layer; nothing is guessed.
The field set the handoff asks for, annotated with the minting actor and the layer that carries it. [PK] fields are authored by the image and folded via the emit [D]; [NF] fields arrive on the execution envelope; [I]/[Y] are the run identity.
Workflow / image identity — Nextflow workflow version + commit [Y]; PKB repository commit [PK]; immutable PKB image digest [Y]; image org.opencontainers.image.revision [PK]; image platform linux/amd64|arm64 [NF]; parent / child trace IDs [I] (run_id + the domain crossing addresses).
Governed run identity — profile_id · release_id · run_id · data_cutoff_date [C/I]; acquisition request/spec sha256; build-template sha256; resolved acquisition-spec sha256 (when available); sealed pkb-build-spec.json sha256 [PK].
Per-source receipt — source ID; adapter ID + acquisition mode; policy / license ID; concrete upstream release/version + release date; retrieval timestamp; canonical origin URL or governed PVC manifest identity; compressed bytes + sha256; uncompressed or deterministic directory-tree sha256; schema / header / profile checks + row counts; receipt sha256 [PK]. (The origin field is the network-vs-local lane; every other field is lane-independent — see The files in play.)
Build / package — preflight result; the exact trusted command plan + per-command exit status; CPU / memory / scratch / thread settings; pre- and post-build input attestation; candidate DuckDB hash + bytes; package-manifest hashes; package-gate report hash + status; PPI stats TSV hash; completion-manifest hash; promotion status + current pointer before/after [PK]. Per-command exit status and the resource settings overlap the execution envelope [NF] — the envelope carries the pod-level truth, the emit carries the image's own record; both fold on run_id and reconcile there.
Never in the trace: credential values, netrc contents, local workstation paths, licensed source rows. Enforced at the emit site — the workflow chooses the payload, so the exclusion is a property of what it passes, not a filter downstream.
The DAG's parallelism is bounded by what the image exposes today:
release_id and candidate roots; CPU-heavy build steps only where the image's command graph and shared DuckDB state support it; and, in future, source-scoped acquisition tasks that each write an isolated workspace and emit a signed receipt, gathered by a single seal.fetch writers against one acquisition root. The current fetch owns a single private attempt, a root lock, an ordered completion record, and atomic publication; concurrent writers violate the recovery and no-partial-publication contract.So the target fan-in is isolated source payload + receipt … → single exact-profile gather / offline seal → immutable release at /inputs, and per-source fan-out stays in WOULD-RUN mode until PKB exposes the source-scoped resolver / materializer interface. When a step is held in WOULD-RUN, that is logged — the trace shows a placeholder, not a silent skip.
The prototype (vivarta-examples/pkb/workflows/pkb/main.nf) proved the surrounding infrastructure; these are the build items that move its shape onto the contract above. The infrastructure carries over unchanged.
| Item | Target |
|---|---|
| the DAG shape | four fixed-interface Jobs (acquire → seal → build → promote) around one image + profile_id, replacing the per-source acquire/parse/upload command graph |
| source fan-out | held in WOULD-RUN until the source-scoped interface lands; single fetch attempt + atomic publish until then |
| promotion | a separate pkb.promote submit, never chained after build |
foxfireEmit SPI |
the domain-emit function (or the generalized foxfire_emit step), fed by the image's authored JSON |
| carries over | governed egress model (deny / controlled labels + standing policies); in-cluster Nexus base-image pull; the signed-crossing PUBLISH path; the dry/real per-step toggle; the execution envelope [NF] |
| Piece | Owner | Depends on | Status |
|---|---|---|---|
| four-Job orchestration in main.nf | platform (this repo) | image entrypoints (Grant) | designed here; prototype realignment pending |
foxfireEmit function extension |
platform (nf-foxfire) | Envelope/Signer/Projection (exist) | built + 18/18 green — FoxfireExtension (@Function foxfireEmit(type:, payload:)) over a shared Emitter/FoxfireRuntime; observer refactored onto the same core, wire output byte-identical |
| domain crossing types in foxfire | foxfire | nothing | :events:pkb:* folds on run_id/event like any crossing; a named boundary-event type is the standing item |
| acquisition-spec / build-template authoring | platform + Grant | profile closures | the governed documents the submit passes |
| source-scoped resolver / materializer | Grant (PKB image) | — | the interface that lifts per-source fan-out out of WOULD-RUN |
| PPI adapter closure | Grant (PKB image) | — | real PPI acquisition fails closed until the 14-handler closure completes |
| governed egress policies | platform (this repo) | exist | cnp-acquire-fqdn-egress, netpol-build-deny-egress — source the FQDN allow-list from the licensed-source policy manifests |
| Item | Fulfilled by | Status |
|---|---|---|
| Nextflow workflow version/commit | [NF] envelope — run:started + run:completed metadata.workflow (revision, commitId, repository, manifestVersion, nextflowVersion) |
✅ observer extended |
| PKB repository commit | image self-reports — emit from pkb-container preflight JSON, or the image's OCI revision label |
⬜ emit preflight · needs image field |
| immutable PKB image digest | [F] placement — the reaper's observed image_id digest; wisp plan also records the driver image |
✅ placement (confirm digest, not tag) |
| image org.opencontainers.image.revision | image OCI label — self-reported in preflight JSON, or resolved from the digest at the registry |
⬜ emit preflight · needs image field |
| image platform (linux/amd64 or linux/arm64) | image self-reports (uname -m) in preflight; or node arch from [F] placement |
⬜ emit preflight |
| parent/child trace IDs | [wisp]/[I] run_id (parent) + the foxfireEmit to addresses (child) |
✅ parent · child from the emit |
| Item | Fulfilled by | Status |
|---|---|---|
| profile_id | [wisp] params-file → run:started params metadata; echoed in every image JSON |
✅ params (confirm it's surfaced) |
| release_id | [wisp] params-file; bound to the sealed manifest by the JB·seal emit | ✅ params · 🔶 bind at seal |
| run_id | [wisp]/[I] — nf-foxfire keys every crossing on it | ✅ core |
| data_cutoff_date | [wisp] params-file → run:started params |
✅ params |
| acquisition request/spec SHA-256 | JA·plan — emit sha of /inputs/acquisition-spec.json (from the image plan JSON, or hashed in the step) |
⬜ emit at plan |
| build-template SHA-256 | JA·plan — emit sha of /inputs/build-template.json |
⬜ emit at plan |
| resolved acquisition-spec SHA-256 when available | JA·resolve — emit from the resolve step's resolved-spec output |
⬜ resolve step + emit |
| sealed pkb-build-spec.json SHA-256 | JB·seal — emit from pkb-build-spec.json (carried in the acquisition-release-manifest) |
⬜ emit at seal |
Every field here lives in the per-source receipt the image writes — retrieval fields at JA·fetch, rehashed and finalized at JB·seal — captured by one foxfireEmit(:events:pkb:source-receipt) per source folding that receipt JSON. One row-shape; the table names where each field originates and which lane fills it.
| Item | Fulfilled by | Status |
|---|---|---|
| source ID | JB·seal receipt (source_id) |
⬜ emit per source |
| adapter ID and acquisition mode | JA·fetch — adapter + network/local mode | ⬜ emit per source |
| policy/license ID | JB·seal receipt — from the profile's source policy | ⬜ emit per source · needs image field |
| concrete upstream release/version and release date | JA·resolve/fetch — the pinned upstream identity | ⬜ emit per source |
| retrieval timestamp | JA·fetch | ⬜ emit per source |
| canonical origin URL or governed PVC manifest identity | JA·fetch — URL (network lane) or /incoming PVC manifest identity (local lane) |
⬜ emit per source |
| compressed bytes/SHA-256 | JA·fetch — over the downloaded payload | ⬜ emit per source |
| uncompressed or deterministic directory-tree SHA-256 | JB·seal — deterministic tree hash of the sealed closure | ⬜ emit per source |
| schema/header/profile checks and row counts | JA·fetch/parse — validation + counts | ⬜ emit per source · needs image field |
| receipt SHA-256 | JB·seal — the image's hash over the finalized receipt | ⬜ emit per source |
| Item | Fulfilled by | Status |
|---|---|---|
| preflight result | JC·preflight — emit from the preflight JSON (+ [NF] exit) |
⬜ emit at preflight |
| exact trusted command plan and per-command exit status | JC·plan (command-plan JSON) + JC·build (per-command exits, image-internal [PK]); pod exit is [NF] | ⬜ emit at plan + build |
| CPU, memory, scratch, and thread settings | pod side [P] wisp plan / overlay (present); image thread settings from the build record | 🔶 pod have · emit image side |
| pre- and post-build input attestation | JC·build — /inputs attestation in the build record |
⬜ emit at build |
| candidate DuckDB hash/bytes | JC·build — from the completion / build manifest | ⬜ emit at build |
| package manifest hashes | JC·build — package-manifest.json |
⬜ emit at build |
| package-gate report hash/status | JC·build (validate_packages) — the gate report |
⬜ emit at build |
| PPI stats TSV hash | JC·build — hash of the stats TSV | ⬜ emit at build |
| completion-manifest hash | JC·build — completion-manifest.json (the natural build-complete crossing) |
⬜ emit at build |
| promotion status and current pointer before/after | JD·promote — from the promote output (the pkb.promote job) |
⬜ emit at promote |
The existing signed trace and nested build-pipeline trace are a good fit. This maps each attribute to the Job · step it is sourced from and how it reaches the signed run.
Capture legend.
TraceObserverV2), automatic, no workflow change. [NF]profile_id, release_id, run_id, data_cutoff_date, pod resources). [wisp]foxfireEmit(type:, payload:) SPI (built, nf-foxfire) folds an image-authored JSON at the named Job·step onto this run_id.Status: ✅ captured today · 🔶 partial (one side present) · ⬜ to build — the observer extension, workflow emit step, or image artifact it waits on (the plugin method is done).