foxfire is the Nextflow pipeline observability receiver — the third tier of the echo reference application. It ingests signed pipeline lifecycle events, stores them append-only, and serves per-task cost and run history over HTTP and GraphQL. It runs on Kubernetes (CoreWeave for dev/prod, OrbStack for local).
POST /events/signed — a signed four-address envelope per event. Verifies each, appends it to an immutable log, folds it into run + task entities, and computes a per-task cost.GET /runs, GET /runs/:run_id, GET /runs/:run_id/tasks, and POST /graphql (run/task/cost, plus the session-lineage field resume resolves through).POST /plan (wisp's per-process plan at launch) and POST /placement (where the reaper saw each pod land); serves GET /runs/:run_id/reconcile (plan-vs-actual drift), /placement/:run_id, /plan/:run_id, and the rolled-up job-type envelope wisp sizes from.POST /admin/refold — re-project a run's entities from the event trail (the read-model repair; see How it works).GET /health for liveness.Nextflow's built-in trace carries observability (timing, resource use, exit codes) but no provenance: events are unsigned, there is no hash chain, and a TraceRecord is a mutable map updated in place. foxfire is the append-only signed store that turns each lifecycle callback into an auditable, billable event with its own timestamp and content hash. It is the receiver half of the Nextflow → platform provenance bridge.
Wanderland::Engine service. Boundaries: ingest / ingest_signed (/events, /events/signed), run / runs / tasks (reads), plan_accept / placement_accept / reconcile_read / envelope (the router-facing surface), graphql_execute (/graphql), refold (/admin/refold), and health.:events: and :pki: mounts run on SQLite locally (fast, on local disk) and on Postgres on dev/prod — both through wanderland-core's storage-driver abstraction, one records table, same record shape. Raw events are append-only; the run and task entities are $merge-upserted as events arrive; reads compose the cumulative view. Postgres is the dev/prod store because the store sat on an NFS-backed volume where SQLite's lock cycle defeated the page cache, re-reading its pages over the network per transaction.replicas: 1), so it knows exactly when the entities change: every write bumps a generation counter, and the hot reads (the runs list, the whole-store task index — the dashboard's 1/s poll) serve from an in-process cache until the generation moves. Exact invalidation, no re-read per request.resume_of submission resolves which session's cache to continue. A cached task reaches the store as status CACHED with no placement.POST /admin/refold re-derives them. When fold-identity rules change under existing history, it regroups a run's events by their true run name, replays the fold, and tombstones the stale entities (append-only store — removal is a flag the readers skip, never a delete).key_id → recompute the payload content hash and compare (422 on mismatch) → verify the EC P-256 signature (401 on bad signature) → fold. The signature, hash, and four addresses ride onto the audit record.advaita/foxfire (ECR 789905347053.dkr.ecr.us-west-2.amazonaws.com/advaita/foxfire), the engine-service image (ruby:3.4-slim + libsqlite3 + libpq; boots wanderland --type http config.yml). Built and pushed by foxfire/vivarta.yml. Platform forced per env — local arm64, dev amd64 (a native build on the wrong arch runs under qemu and pays ~5× CPU per request).replicas: 1. Container port 9311 (http). Readiness + liveness on /health. Single writer is a design invariant — the generation-cached read model lives in process, so scaling past one replica means moving the read model out of process first.foxfire-data, RWO, mounted at /data; FOXFIRE_DB=/data/foxfire.db). Dev runs Postgres — a single-replica foxfire-pg StatefulSet in vivarta-echo, one database with events and pki schemas (each mount scopes itself via search_path in its connection URL; the seal-chain keys are restart-safe again). Prod is CoreWeave managed Postgres — a DSN swap in the overlay, no pod. Selected by FOXFIRE_DB_DRIVER / FOXFIRE_DB.whitelist-source-range: 152.233.5.238/32) on dev/prod; TLS via cert-manager (letsencrypt-prod). local is plain HTTP with no whitelist.foxfire/cfn/dns.yml) pointing the host at the cluster's nginx-ingress IP, provisioned in the same deploy (AWS DNS + k8s workload in one component iterator, DNS first so cert-manager's HTTP-01 resolves). local omits it.vivarta-echo.| Env | Context | Host | Store | Image stage |
|---|---|---|---|---|
| local | orbstack |
foxfire.localhost |
SQLite (PVC) | build → ECR (arm64) |
| dev | useast13-dev |
foxfire-dev.cw.advaitabio.com |
Postgres (foxfire-pg) |
build → ECR (amd64) |
| prod | useast13-prod |
foxfire.cw.advaitabio.com |
Postgres (CW managed) | promote (k8s_live_image, no rebuild) |
config.yml: service/port/boundary_path, the :events: SQLite mount (FOXFIRE_DB, default data/foxfire.db), and keys: — the public verification keys by key_id.keys: currently ships {}. A deployed receiver with keys: {} returns 401 unknown key_id for every signed event, so it must carry the matching public SPKI PEM under keys: { <key_id>: <pem> } before signed ingest verifies. Provisioning path is open (see Open questions).foxfire/vivarta.yml (containerized_deployment; docker_build → ECR; k8s_provisioner; plus the CFN DNS component on dev/prod).vivarta-drones/scenarios/apps/foxfire/foxfire-deploy-local.yml scripts the OrbStack-local deploy: rsync wanderland-core and the three :deploy packs to $WORKSPACE siblings (checkout-extra, real copies — Docker won't follow a symlink), docker build the receiver tagged to the local overlay's image, kubectl --context orbstack apply -k k8s/overlays/local, then wait for the rollout. The receiver image builds and boots on OrbStack — a running container serves /health ({"ok":true,"service":"foxfire"}).
Local image pull mirrors CoreWeave at the k8s-contract level: an ecr-creds dockerconfigjson secret in vivarta-echo, attached to the namespace default ServiceAccount. On CW an ESO ExternalSecret + ECRAuthorizationToken generator mints it hourly; locally vivarta-platform/local-k8s/ecr-pull/refresh-ecr-creds.sh mints the same secret from the developer's AWS credentials (~12h token) and default-sa-pull-secret.yaml attaches it. With imagePullPolicy: IfNotPresent, a locally-built image tagged as the ECR ref runs without a pull; ecr-creds covers the real ECR pull path.
| Dependency | Use | Endpoint / id |
|---|---|---|
| ECR | image registry | advaita/foxfire |
| Route53 (dev/prod) | public DNS A record | zone Z09954442XO6C5WWVED2R (cw.advaitabio.com) |
| cert-manager (dev/prod) | TLS issuance | letsencrypt-prod ClusterIssuer |
| SSM (planned) | public verification key | path: TBD |
| Dependent | Use |
|---|---|
| echo-backend | proxies GraphQL reads (foxfire_graphql); points launched driver Jobs at /events/signed |
| nf-driver | the driver's nf-foxfire plugin POSTs signed events to /events/signed |
| wisp | reads the rolled-up job-type envelope to size a run; posts the plan back at launch |
| reaper | posts placement (where each pod landed) for plan-vs-actual reconcile |
| puck | the launched sorties read run/reconcile/placement to verify their submissions |
No new run/task telemetry is stored; workflow execution is unaffected (the driver runs regardless). The echo dashboard shows stale or empty data, and provenance/billing has a gap for the outage window. Whether the driver's shipper queues, drops, or fails the run when the receiver is unreachable: TBD. Blast radius is the observability plane, not the compute plane.
foxfire (git@git~graemefawcett/foxfire, initial commit d069ef9). Ruby / wanderland-core engine. docs/event-envelope.md is the wire contract; driver/ holds the separate nf-driver image; k8s/ and vivarta.yml are the deploy.config.yml keys: {} needs the SPKI PEM by key_id from SSM; the SSM key-provider in ingest_signed#resolve_public_key is not implemented (it reads config["keys"] only): TBD.keys: (mounted ConfigMap / env-substituted config): TBD.Certificate Ready) on useast13-dev / -prod: TBD.foxfire-data PVC: TBD.166.19.90.28 / prod 166.19.90.12) shared with echo-frontend: TBD verify.!Env FOXFIRE_SEAL_KEY + auto-gen (currently persisted on the PVC) + a single service:foxfire identity onto the storage-backed :pki: path provisioned by !SsmLookup (private SecureString under the per-env CMK; public half published), using a history-walking verifier so old chains stay verifiable across rotation. Depends on the wanderland-core Activities::PKI change (verify walks key history + a rotate verb + demote scrubs private_pem). Also: data/foxfire-pki.db + data/foxfire.db are committed dev fixtures — gitignore and purge on externalization. Standard: advaita-key-management-standard; core: wanderland-core-pki; case task-55de866f (PKI-EXTERNALIZE).