Advaita Service: nf-driver

nf-driver is the compute plane of the echo reference application — the Nextflow driver image, launched on demand as a one-shot Kubernetes Job by echo-backend. The driver runs the fibfizz workflow with the Kubernetes executor, spawning a task pod per process, and its baked-in nf-foxfire plugin signs each lifecycle event and ships it to foxfire. It is built and pushed to ECR but never deployed as a long-lived workload.

What it does

• Runs nextflow -c …/nextflow.config run …/main.nf -profile kubernetes --x N as a Job.
• Under the k8s executor, each process (FIBONACCI, FIZZBUZZ) becomes its own task pod (python:3.11-slim), sharing the Nextflow work dir on an RWX PVC.
• The nf-foxfire TraceObserver, running in the driver process, signs each event (EC P-256) and POSTs it to the receiver's /events/signed.

Why it exists

It is the workload the launcher fires and the receiver observes — the async compute plane that produces provenance and billing data. Only the driver holds the signing key; task pods never see it and never call foxfire.

How it works

• Driver image advaita/nf-driver: eclipse-temurin:21-jdk + Nextflow 26.04.4 + the baked fibfizz workflow + the nf-foxfire plugin at $NXF_HOME/plugins` (`NXF_OFFLINE=true`, so a no-egress driver pod loads it from disk). Built by `foxfire/driver/vivarta.yml` (image-only: `deploy: { type: noop }`). The plugin and wanderland path-gems are checked out as `$WORKSPACE siblings and COPYed, per the vivarta checkout-extra convention.
• Launch: echo-backend's launch_run renders the Job — driver image, nextflow-runner SA, nextflow-work PVC at /work, ttlSecondsAfterFinished, and the three FOXFIRE_* env vars. Fire-and-forget; foxfire tracks progress.
• Task pods: stock python:3.11-slim (process.container), spawned by the driver via the k8s executor; they do the work, write the shared PVC, and never call foxfire.
• nextflow.config: plugins { id 'nf-foxfire@0' } + foxfire { url = env FOXFIRE_URL; keyId = env FOXFIRE_KEY_ID }; the observer gates on FOXFIRE_URL.

Deployment (k8s)

• Not a deployed workload. Built and pushed to ECR (advaita/nf-driver); launched as batch/v1 Jobs into vivarta-echo.
• Execution substrate (separate infra track), applied once per cluster:- foxfire/k8s/infra/nextflow-runner.yaml — nextflow-runner ServiceAccount + Role (create/watch/delete pods + PVCs) + RoleBinding, for the k8s executor to spawn task pods.
• foxfire/k8s/infra/nextflow-workdir-pvc.yaml — nextflow-work, a ReadWriteMany PVC mounted /work across the driver and task pods; storageClass is cluster-specific (RWX-capable).
• Build env is dev (image-only); the tag is a git sha.

Configuration

• foxfire/workflows/fibfizz/nextflow.config — the nf-foxfire plugin block + the k8s executor profile (process.executor=k8s, process.container=python:3.11-slim, k8s { namespace / serviceAccount / storageClaimName / storageMountPath=/work }, workDir=/work/work).
• Injected at Job time by the launcher: FOXFIRE_URL=<receiver>/events/signed, FOXFIRE_KEY_ID, FOXFIRE_SIGNING_KEY (EC P-256 PKCS#8 PEM, via secretKeyRef to foxfire-signing-key, provisioned from SSM).

Local build

vivarta-drones/scenarios/apps/foxfire/nf-driver-build.yml scripts the OrbStack-local build: rsync the nf-foxfire working copy to $WORKSPACE/nf-foxfire (the checkout-extra sibling, a real copy since Docker won't follow a symlink), then docker build -f driver/Dockerfile. The plugin is pinned nf-foxfire@0 in nextflow.config. Validated via bin/drone run: the image builds, the plugin bakes into /opt/nextflow/plugins, and Nextflow loads it offline (NXF_OFFLINE=true) — proven with a trivial in-container pipeline that loads the plugin without a registry fetch.

Dependencies

Dependency	Use
foxfire	signed event sink (/events/signed)
nf-foxfire plugin	JVM emit plugin baked into the image
nextflow-work RWX PVC	shared Nextflow work dir
nextflow-runner SA	spawn task pods
Kubernetes API	create task pods (via nextflow-runner)
ECR	image registry (advaita/nf-driver)

Dependents

• echo-backend launches it. foxfire consumes what it emits.

Impact when down

A missing or unbuildable image leaves launched runs pending (ImagePullBackOff) — the piece the fe/be/foxfire bring-up waits on. A failed driver Job yields no telemetry for that run; other runs and the standing tiers are unaffected.

The code

• Driver image + workflow: foxfire repo (driver/Dockerfile, driver/vivarta.yml, workflows/fibfizz/). Plugin: nf-foxfire (git@git~graemefawcett/nf-foxfire), JVM/Gradle, TraceObserverV2. The driver image is owned by the nf-foxfire workstream (case task-263e21e1).

Open questions

• Shared Nextflow work dir: OrbStack (single node) uses an RWO local host volume; CoreWeave (multi-node) uses an RWX storage class for nextflow-work.
• Signing-key provisioning: the local-bootstrap drone generates the EC P-256 keypair and creates the foxfire-signing-key (private, PKCS#8) + foxfire-public-key (public, SPKI) Secrets per cluster.
• Jenkins build: jcasc/projects.yml needs an nf-driver entry with extra_repos landing nf-foxfire at $WORKSPACE/nf-foxfire (handoff to vivarta-jenkins).
• file.publish / workflow.output custody events: deferred.