How a run's trace lands in the store, written as a ledger: the one record shape everything writes, a worked run showing every record type with its full payload, where each field is sourced, and — at the end — the diff against what the code does today.
Sibling of advaita-nextflow-router-data-model (Stage G — the ledgers) and advaita-nextflow-driver-data-model (D5/D6 — where the producer emits). This page is the storage-record view those two only gesture at: what a single row actually is.
Worked against the real run echo-20260803115844-589cd7 (echo.fibfizz, x=20) — the rows
sitting in foxfire-dev events.records as of 2026-08-03, so it diffs straight against code.
A record is three routing fields and an opaque body. Where it's going, where it came from, what type it is — plus a payload we don't structure. Everything in the system writes this shape; record types differ only in the payload, never in the columns. The store owns routing, sealing, and folding. The producer owns "something happened."
┌─ routing (structured, indexed) ─┬─ chain (structured) ─┬─ body (opaque) ─┐
│ to from type │ trace signature │ payload │
└─────────────────────────────────┴──────────────────────┴─────────────────┘
| field | meaning | filled by | notes |
|---|---|---|---|
to |
the entity this record is about | store [S] |
:events:run:<run> / :events:task:<run>:<hash> / :events:manifest:<run> / :events:file:<run>:<name> |
from |
the source it came from — a real lineage edge | store [S] |
plan→run→task→file; never = to. from → to is the causal DAG (see the worked run) |
type |
the record type | store [S] |
:events:run:started, :events:task:complete, … — the real type, queryable in SQL |
trace |
pointer to the previous record's signature in this scope | store [S] |
the chain link; the sequence partitions by run_id, not from |
signature |
one signature over the body | producer [P] (optional) |
the emitter's, if we don't trust the channel — not re-computed by the store |
payload |
the body | producer [P] |
opaque JSON. Parse it if you want. Could be Mongo. |
at |
event time | producer [P] |
full precision (utc_time), the ordering key inside a scope |
Structured columns: 7. Everything else is body. No boundary, capabilities,
requirements, result, scope, crossing_id, source_boundary — those are dead weight
(all null on every foxfire event today).
| Tag | Source | Owns |
|---|---|---|
| [N] | Nextflow | the raw fact — lifecycle transitions, the resource trace, workflow meta |
| [P] | nf-foxfire plugin (the producer, in the driver pod) | stamps run identity onto the fact; emits it; optionally signs the body |
| [W] | wisp | launch context handed to the driver as env / params-file (run_id, job_type, params) |
| [I] | platform identity | the minted run_id, the Nextflow session_id |
| [S] | foxfire store (receive side) | assigns to/from/type, the chain trace, the read-model fold, computed cost, seq |
The load-bearing claim of this whole page: to/from/type/trace are [S]. The
producer never mints an :events: address or a chain link. It says what happened; the store
files it.
echo-20260803115844-589cd7Six event types, in order — and read together they are a causal DAG, from → to:
from read top-to-bottom is the lineage: wisp → plan → run → tasks → files. <run> =
echo-20260803115844-589cd7.
| # | to | from | type | payload (essence) |
|---|---|---|---|---|
| 1 | plan:<run> |
wisp/classify |
:events:plan |
echo.fibfizz · sm · workers-immediate-cpu · per_process{FIB,FIZZ} [W] |
| 2 | run:<run> |
plan:<run> |
:events:run:started |
params{x:20} · session fe70c487 · nf 26.04.4 |
| 3 | task:<run>:78/a67bda |
run:<run> |
:events:task:submit |
FIZZBUZZ · SUBMITTED · attempt 1 |
| 4 | task:<run>:6b/deb195 |
run:<run> |
:events:task:submit |
FIBONACCI · SUBMITTED · attempt 1 |
| 5 | task:…:78/a67bda |
run:<run> |
:events:task:start |
RUNNING · start 11:59:03 |
| 6 | task:…:6b/deb195 |
run:<run> |
:events:task:start |
RUNNING · start 11:59:03 |
| 7 | task:…:78/a67bda |
run:<run> |
:events:task:complete |
COMPLETED · exit 0 · 2022ms · rss 11.5MB |
| 8 | task:…:6b/deb195 |
run:<run> |
:events:task:complete |
COMPLETED · exit 0 · 3024ms · rss 10.5MB |
| 9 | run:<run> |
plan:<run> |
:events:run:completed |
success · 15607ms · exit 0 |
| 10 | file:<run>:trace-…txt |
run:<run> |
:events:file:published |
nf trace report · 705B · run-level (no task) |
| 11 | file:<run>:6b/…/fibonacci.txt |
task:…:6b/deb195 |
:events:file:published |
64B · sha 158822… · "0 1 1 2 3 5 8…" |
| 12 | file:<run>:78/…/fizzbuzz.txt |
task:…:78/a67bda |
:events:file:published |
77B · sha cdfaef… · "1 2 Fizz 4 Buzz…" |
| 13–28 | file:<run>:<hash>/.command.* |
task:…:<hash> |
:events:file:published |
×16 more (.command.sh/run/out/err/log, .exitcode) — each FROM its task |
| 29 | manifest:<run> |
run:<run> |
:events:manifest:complete |
published 19 · failed 0 |
29 records for the run (1 plan + 1 run-start + 2×3 task + 1 run-complete + 19 files +
1 manifest) — versus 57 today, and none are $merge pokes: the run/task/manifest
entities are the fold of these, materialized separately (the read model). Because from is
lineage, SELECT to FROM records WHERE from = ':events:task:<run>:6b/deb195' AND type = ':events:file:published' is "everything FIBONACCI produced", and walking from backwards is
the origin backtrace in SQL.
:events:plan:<run> ──▶ :events:run:<run> run started FROM its plan
:events:run:<run> ──▶ :events:task:<run>:<h> task submitted/ran FROM the run
:events:task:<run>:<h> ──▶ :events:file:<run>:<n> file published FROM the task that made it
from is a real edge, never = to. The chain/sequence partitions by run_id (embedded
in every address), so from is free to carry lineage — and the file→task edge is the driver
node's origin backtrace as a routing column, subsuming from_input. Each block below is one
row; payload is lean — only what the producer knows.
to :events:run:echo-20260803115844-589cd7 [S]
from :events:plan:echo-20260803115844-589cd7 [S] ← the plan launched it (not self)
type :events:run:started [S]
trace <genesis> [S] first in scope
signature MEUC… (over payload) [P] optional
at 2026-08-03T11:58:58.526324120Z [P] full precision
payload:
run_name: echo-20260803115844-589cd7 [I]
session_id: fe70c487-20b9-4a8d-b695-3b28941338fd [I/N]
job_type: echo.fibfizz [W]
params: { x: 20 } [W] ONE copy, correct type
workflow: { projectName: main.nf, nextflowVersion: 26.04.4, scriptId: a68f3ebd… } [N]
to :events:task:echo-…589cd7:78/a67bda from :events:run:…589cd7 type :events:task:submit [S]
payload:
trace: { task_id: 1, process: FIZZBUZZ, tag: fizzbuzz-20, hash: 78/a67bda,
native_id: nf-78a67b…, status: SUBMITTED, submit: 1785758339170,
attempt: 1, cpus: 1, memory: 536870912 } [N]
from = the run — the run submitted the task. (Already right in the producer data.) hash/
process/task_id live only inside trace; the store lifts what it folds.
to :events:task:…:78/a67bda from :events:run:… type :events:task:start [S]
payload: { trace: { …as submit…, status: RUNNING, start: 1785758343892 } } [N]
Only the delta the producer observed. The store composes cumulative state; the producer does not carry the fold.
to :events:task:…:78/a67bda from :events:run:… type :events:task:complete [S]
payload:
trace: { …, status: COMPLETED, exit: 0, complete: 1785758343000,
realtime: 2022, duration: 3830, peak_rss: 11546624, rss: 11546624,
"%cpu": 1.5, vmem: 17543168, read_bytes: 0, write_bytes: 45056, … } [N]
Cost is not here. cost is [S] — the store prices the trace at fold time onto the task
entity, never back onto the raw event. The producer reports usage; the store prices it.
to :events:run:…589cd7 from :events:plan:…589cd7 type :events:run:completed [S]
payload: { workflow: { success: true, exitStatus: 0, duration: 15607 } } [N]
Bookend of the plan→run lifecycle (from = plan — the launch source; or the driver — never
self). The run seal; with trace-offload on, the file tail lands after it, so "last event" ≠
completion — the verifier judges the seal (advaita-nextflow-router-data-model §Stage G).
Per file (one row each, at its own address):
to :events:file:echo-…589cd7:work/6b/…/fibonacci.txt [S]
from :events:task:echo-…589cd7:6b/deb195 [S] ← FROM the task that made it (lineage)
type :events:file:published [S]
payload:
name: work/6b/deb1956…/fibonacci.txt [N]
size: 64 [N]
sha256: 15882217a780… [N]
content: "0 1 1 2 3 5 8 13 21 34 …" (or {_iou} ref if large) [N]
from = the producing task is the origin backtrace as a column: walk WHERE from = :events:task:… to every file a task made; walk a file's from back to its task, the task's
from back to the run, the run's from back to the plan. from_input (driver §D5) stops
being a special payload field — it is from. Terminal drain signal (one row):
to :events:manifest:… from :events:run:… type :events:manifest:complete [P→S]
payload: { published: 19, failed: 0 } [N]
The manifest is a read-model projection — a scan/count over the run's
:events:file:published rows — not a stored growing array (today it is a 19-poke chain →
O(N²); see the diff).
[S] — never producer-authored):projections:* rows are the fold — the store's materialized view of the trace, for
reads. They are [S], never producer-authored. Two rules, straight from the projection design:
run:completed arrives it triggers the fold: the store
computes the run + task rollups once and writes them as records. From then on every
consumer reads the sealed record — nothing re-folds.:projections:run:<run> yet is still
running; its fold is computed in memory from the raw events as a fallback. The
determination is one lookup — sealed record present → read it; absent → live-fold — so memory
holds only what is in flight.Three projection records, each backing one offered interface (work backwards from the API):
:projections:task:<run>:<hash> — type :projections:task, from :events:task:<run>:<hash>.
The billing grain, sealed at task:complete. Cost is computed here ([S]), never on the event.
:events:plan:<run> is not just wisp's sizing decision — it carries the whole submit
(router §Stage A/B), because the caller + tenancy + inputs are needed by everything downstream
(cost attribution above all). Its from is the caller app, so the lineage roots at who
asked: app → plan → run → task → file.
to :events:plan:<run>
from :app:<app_id> ← the caller (roots the DAG; settles the bookend thread)
type :events:plan
payload:
app_id: echo-frontend # [C] who called (crossing identity app:<app_id>)
run_id: <run>
job_type: echo.fibfizz # [C]
immediacy: immediate # [C]
tenancy: { org_id: echo, team_id: echo, owner_id: echo-launcher } # [C] user triple — ALWAYS present
params: { x: 20 } # [C] the knobs
data: { } # [C] input map (empty for fibfizz; hub://… refs when present)
size: sm · compute: cpu · queue: workers-immediate-cpu # [P] wisp decision
driver: { image, namespace, queue } # [P]
per_process: { FIBONACCI: {…}, FIZZBUZZ: {…} } # [P] sizing + p95 evidence
Fibonacci has no data (no inputs), but the identifiers are always there — app_id +
tenancy — so every run is attributable regardless of workload.
:projections:cost:<run> — type :projections:cost, from :projections:run:<run>. The
billing extract: who + how many credits, keyed by run_id. This is the record that feeds the
future billing/credits system — that system holds its own run info by run_id and always comes
back to this store for the trace (one verify away).
payload: { run_id, app_id: echo-frontend,
tenancy: { org_id: echo, team_id: echo, owner_id: echo-launcher },
job_type: echo.fibfizz, credits: 0.005607, compute_ms: 5046, overhead_ms: 10561,
per_process: { FIBONACCI: 0.00336, FIZZBUZZ: 0.002247 } }
The fold reads app_id/tenancy from the plan + credits from the run rollup and writes it at
seal. Adds to the projection set: :projections:cost (per run). run_id is the spine the
downstream system joins on; the trace stays here.
cost field above)No calculated dollar cost. Track CPU time and bill in credits: 1 credit = 15 minutes of CPU time (900 CPU-seconds). And split compute from the overhead of running on pods.
Process time vs wall time — what the Nextflow trace gives us (task:complete, already on the wire):
realtime = start → complete = compute time (the process). Excludes pod spin-up & teardown.duration = submit → complete = compute plus scheduling + pod-startup overhead.complete is process-end;
capturing it needs the reaper to record pod-terminate time. The one gap.The fold computes, at the task grain (replacing cost):
compute: { cpus: 1, compute_ms: 3024, # realtime — process time
cpu_ms: 3024, # cpus × compute_ms — billable CPU-time
credits: 0.00336 } # cpu_ms / 900000 (15 CPU-min = 1 credit)
overhead: { overhead_ms: 2756, # duration − realtime (schedule + pod startup)
teardown_ms: null, # NOT MEASURED — needs reaper pod-terminate
overhead_pct: 0.48 } # overhead_ms / duration
Rolled up on the run:
compute: { compute_ms: 5046, cpu_ms: 5046, credits: 0.00561 }
overhead: { wall_ms: 15607, overhead_ms: 10561, overhead_pct: 0.68 } # wall − compute (driver + tail)
Open decision (yours): credits bill on compute (realtime) only — overhead is tracked
but not billed, so the platform eats spin-up and the split stays visible; overhead_ms is
already recorded if you later want to bill it. On dev the overhead % is inflated (tiny tasks,
cold pods); real runs shrink it. When you want the finer cut, placement (created/scheduled/
started timings) splits overhead_ms into queue-wait vs pod-startup; teardown stays a gap
until the reaper records pod termination. Source: [S] computes this at fold from the event's
realtime/duration + placement — the same seam as the old cost().
Walked back from all 14 read endpoints (config.yml). Every offered field is pre-computed
into a record; only the raw-provenance surfaces drop to :events:.
| endpoint | consumer | reads |
|---|---|---|
GET /runs |
UI | :projections:run:* |
GET /runs/:id |
UI | :projections:run + :projections:task:* + events[] raw |
GET /runs/:id/tasks |
UI | :projections:task:* (carries planned + placement) |
GET /runs/:id/manifest |
UI | :projections:manifest:<run> (+ per-run file scan) |
GET /runs/:id/reconcile |
UI · drift | :projections:reconcile:<run> |
GET /runs/:id/otlp |
UI · telemetry | :projections:otlp:<run> |
GET /runs/:id/file |
UI | :events:file:published — raw bytes |
GET /runs/:id/verify |
UI | :events:* seal chain — raw |
GET /runs/:id/chain |
UI | :events:* seal chain — raw |
GET /envelope/:job_type |
wisp | :projections:envelope:<job_type> |
GET /plan/:run_id |
wisp | :events:plan:<run> (first-class) |
GET /placement/:run_id |
wisp · UI | :events:placement:<run>:<pod> (first-class) |
POST /graphql runs/run |
UI | :projections:run + :projections:task (+ Event/events raw) |
GET /health |
probe | run count |
The full record set
[S]: :projections:run, :projections:task,
:projections:manifest, :projections:reconcile, :projections:otlp (per run) ·
:projections:envelope (per job_type).:events:plan:<run> (wisp @ /plan), :events:placement:<run>:<pod> (reaper @ /placement).:events:event/run/task/file, read by
events[], verify, chain, file, and GraphQL Event.Joins folded once, at seal (so the read is pure):
:projections:task carries per-process planned (from the plan) + placement (from the
reaper), so /tasks needs no join at read.:projections:reconcile is the drift verdict — planned vs placed vs used per process —
computed at seal from plan · placement · trace (today reconcile.rb joins on every read).:projections:run.processes and :projections:envelope both derive from the task rollups:
exact p95 per run, merged across runs for the envelope.Run/Task/Cost resolve straight off :projections:*; only Run.events / the
Event type drop to raw — that's "most GraphQL points at record types."Every field from the audit is covered. The only compute-on-read that remains is a single
in-flight run's live fold (the fallback) and a bounded per-run file scan for manifest.files[].
payload: { process: FIBONACCI, name: "FIBONACCI (fib-20)", status: COMPLETED, exit: 0,
cpus: 1, realtime_ms: 3024, peak_rss_bytes: 10493952,
read_bytes: 0, write_bytes: 45056, cost: { total: 0.0000333, currency: USD } }
→ backs GET /runs/:id tasks[], GraphQL run.tasks.
:projections:run:<run> — type :projections:run, from :events:run:<run>. The completed-run
record — sealed when run:completed fires; the thing wisp / GraphQL "grab" for metrics.
payload: { run_id, run_name, job_type: echo.fibfizz, status: succeeded,
started_at, completed_at, duration_ms: 15607, exit_status: 0,
session_id, retries: 0, task_count: 2, total_cost: 0.00005557,
processes: { FIBONACCI: { samples: 1, cpus: 1, realtime_ms: 3024,
peak_rss_bytes: 10493952, cost: 0.0000333 },
FIZZBUZZ: { samples: 1, cpus: 1, realtime_ms: 2022,
peak_rss_bytes: 11546624, cost: 0.00002227 } } }
→ backs GET /runs + GraphQL runs (status / task_count / total_cost per row), the run
headline, and feeds the envelope below.
:projections:envelope:<job_type> — type :projections:envelope, from :projections:run:*.
Cross-run p95 for wisp sizing, refreshed when a run of the job_type seals (merges the per-run
processes above — exact p95 per run, merged across runs).
payload: { job_type: echo.fibfizz, n_runs: 16, source: "sealed rollup",
processes: { FIBONACCI: { samples: 16, fanout_per_run: 1.0, cpus: 1, gpu: 0,
realtime_ms: { mean: 3450, p95: 3458 },
peak_rss_bytes: { mean: 44951689, p95: 45113344 } },
FIZZBUZZ: { samples: 16, fanout_per_run: 1.0, cpus: 1, gpu: 0,
realtime_ms: { mean: 2452, p95: 2475 },
peak_rss_bytes: { mean: 44700467, p95: 46157824 } } } }
→ backs GET /envelope/:job_type (the router's Stage C sizing input).
Read paths, by offered interface:
| offered | today | sealed source | in-flight fallback |
|---|---|---|---|
GET /envelope/:job_type (wisp) |
full-store scan, TTL-cached | :projections:envelope:<job_type> |
recompute from run rollups |
GET /runs · GraphQL runs |
fold in RAM per read | SELECT … :projections:run:* |
live-fold active runs only |
GET /runs/:id · GraphQL run |
fold in RAM | :projections:run + :projections:task:* |
live-fold if active |
GET /runs/:id/verify |
walks the chain | — (always the raw event chain) | — |
The event stream stays authoritative; the projections are its sealed view, and verify
judges the events, never the fold. This is advaita-projection-data-model made concrete —
and it is the memory fix (case task-69fb6d77): nothing whole-store ever enters RAM, because the
only thing folded in memory is a single in-flight run.
Read a field, walk to its origin:
payload.params.x = 20
└─ [W] wisp params-file → driver env → Nextflow `params` (router §Stage B, params.json)
payload.job_type
└─ [W] wisp classify, from the template `when:` match (router §Stage D)
payload.session_id
└─ [I/N] Nextflow session (resume lineage; router §Stage E)
payload.trace.* (realtime, peak_rss, exit, …)
└─ [N] Nextflow task execution → nf-foxfire TraceObserver (driver §D6)
payload.sha256 / from_input (storage runs)
└─ [N/C] driver: computed on PUT, witnessed by the bucket (driver §D4–D5, the origin backtrace)
to / from / type / trace / signature-verification
└─ [S] foxfire receive side — assigned/verified on ingest (THIS is what must move off the producer)
cost
└─ [S] foxfire fold — priced from the trace (event_store.rb `cost`)
run_id
└─ [I] wisp `submit` mints it once — the join key for every ledger (router §Stage B)
The spine is run_id. The producer stamps it; the store routes on it; every ledger
(plan / placement / trace) joins on it. Everything else is either a Nextflow fact [N] or a
launch fact [W] — nothing addressing-shaped originates at the producer.
What the real rows do that this model says they shouldn't. (All observed on
echo-20260803115844-589cd7; code refs are lib/foxfire/event_store.rb unless noted.)
| # | Today | Target | Where |
|---|---|---|---|
| 1 | Producer mints to/from/type and ships them inside payload; row columns are to_addr=:events:event:<run>, from_addr=boundary:ingest, type_addr=:events:event (generic) |
Producer sends a domain fact; store assigns real to/from/type to the columns |
fold_one hardcodes event_cell / FROM_ADDR / EVENT_TYPE |
| 2 | Two signatures: producer ECDSA in payload.signature and a store RSA head-seal in the signature column |
One signature (the producer's) in the column; chain via trace = prev record's signature |
head-seal mode on the :events: mount re-signs every row |
| 3 | Chain scoped per synthetic cell (:events:event:<run>), forcing the log-cell to_addr |
Chain scoped per run via from (:events:run:<run>), so to stays the real target |
unique(to_addr, trace) head-seal |
| 4 | 7 columns always null (boundary/capabilities/requirements/result/scope/crossing_id/source_boundary) |
Drop them — record is 3 routing + 2 chain + payload + at |
schema ensure_schema (wanderland-core) |
| 5 | at truncated to seconds; nanosecond utc_time unused → dense bursts order by insert-id |
Order by full-precision event time | at column granularity |
| 6 | params arrives double-nested + mistyped: params.params.x="20" (str) and .params.params.x=20 (int); run_id ×3 |
One params: {x:20} |
producer bug (nf-foxfire / wisp params-file) |
| 7 | Manifest = 19-poke chain, each poke re-writes the whole files array → 190 entries / |
Files are their own rows (:events:file:published); manifest = a scan/count |
fold_manifest |
| 8 | Task metrics stored ~4×: raw event payload.trace → task poke value.trace → task flat fields → .command.trace file |
Trace once on the event; store lifts what it folds | fold_task copies trace + flattens |
| 9 | Read model = $merge-poke chains re-composed in RAM on every read |
Materialized head rows / capability aggregates (Lever 2) | tasks_index, list_runs, each_overlay |
The through-line: rows 1–3 are all one root cause — head-seal (a generic, unsigned,
single-cell sealer) bolted onto pre-signed, domain-addressed producer events. Fix that and
the payload-stuffing, the double signature, and the log-cell to_addr all collapse into "the
record simply is the crossing." Rows 6–8 are independent producer/fold bugs worth killing
regardless. Row 9 is the memory work (Lever 2) this whole investigation started from.