How an append-only event ledger becomes a cheap, queryable read — written as a ledger of shapes: what a projection is as data, where its output lives, how it's realized (native SQL or Ruby), and when it's recomputed. This is the correctness gauge. The engine that consumes these shapes is advaita-projection-engine; the shapes are the contract.
Sits under advaita-nextflow-router-data-model §Stage G — which names the three authoritative ledgers (plan · placement · trace) joined on run_id — and on wanderland-core-storage-engine — the append-and-read-by-prefix substrate (to/from/type addrs, overlay via antiparticles, signed-from, IOU blob extraction). A projection reads those ledgers and writes one more record shape back.
Two things ship today; a third is the target the first two grow into. Read every later section against this split.
Storage::Registry.capability, realized native per driver (cap_<name> on the Postgres and sqlite drivers) or through the shared in-memory Storage::Capabilities fallback. Three capabilities: list, aggregate, pending. The fallback is the reference semantics; a parity spec asserts native equals fallback on a shared seeded store.:projections: row per completed run from config-declared triggers on the terminal crossing, and its reads resolve off those sealed rows unioned with a live compose for in-flight runs. There is no generic projection engine underneath — each projection is a named Ruby boundary wired in config.yml.over / partition / reduce / into / freshness / seal_when), a native-verb reduce: vocabulary with exact p95, a watermark, /reload/projections, and refold-as-generation. Sections marked [target] below describe this. It is not built. foxfire's trace_otlp marks itself a stopgap until it lands (cases task-7d58f481 and task-06fe434c).A projection is a memoized fold, addressed. It invents no new primitive: it is a reduction over records the store already holds, whose result is written back as one more record under :projections:, and whose recomputation is deferred until a query forces it — burn the CPU iff you need it. The fold itself is either set algebra the mounted driver runs natively, or a registered Ruby reducer for judgments SQL can't express. Everything downstream is a derived view that reconciles to the ledger (lebowski-corollary); a projection is a cache of such a view, and is always rederivable (the wanderland-core-storage-engine refold proves it).
Two seams realize the idea today. The capability surface computes the fold once, portably — native or fallback. foxfire's seal triggers memoize the fold — they write the computed result back as a :projections: row when the run closes. The declarative engine that would unify wiring and memoization is the target.
| Tag | Who computes the fold | When it applies |
|---|---|---|
| [N] | Native — the mounted driver, in-backend (Postgres: jsonb ops, GROUP BY, keyset LIMIT) |
metrics / set algebra |
| [R] | Ruby fallback — the in-process reducer over overlay_all. This IS the reference semantics — [N] is property-tested equal to it |
everywhere native is absent (sqlite / :memory: / local), and as the correctness oracle |
| [B] | Boundary — a registered Ruby boundary (verify_run, the rollup_* seals) that joins one or more ledgers and reasons |
composite judgments (verdict / drift / findings) and the seal-time rollups |
[N] and [R] must agree by construction; [B] is Ruby-only (a judgment has no SQL form and forcing one re-grows the crossings planner the little sister deliberately dropped).
Six record families carry the model; nothing else crosses a boundary. A projection adds rows under :projections:, never a new store.
| Address | Written by | Type | Carries |
|---|---|---|---|
:events:event:<run_id> |
ingest (single writer) | :events:event |
the immutable signed audit trail at one head-sealed cell |
:events:run:<run_id> |
fold_run (on ingest) |
:events:run |
the composed run entity (status, timing, retries, session) |
:events:task:<run_id>:<hash> |
fold_task (on ingest) |
:events:task |
the composed per-task entity (the billing grain) |
:events:<run_id>:plan |
wisp classify | :plan |
the sizing decision + the p95 evidence it was made from |
:events:<run_id>:placement:<hash> |
the reaper | :placement |
where each task actually landed |
:projections:<run_id>:<kind> |
a rollup_* seal boundary |
:projections:<kind> |
the memoized fold for one run |
The first five carry the raw ledgers (advaita-nextflow-event-system fills the trace ledger; event_store.rb folds run/task). The sixth is the built projection family, addressed run-major: the run id sits in the leaf-2 slot and the kind is the leaf, so :projections:<run>:run, :projections:<run>:task:<hash>, :projections:<run>:reconcile. "Every run of a kind" is therefore a type scan (list filtered by type_addr), not a prefix scan.
[target] The declarative engine addresses role-major instead — :projections:<name>:<partition>, partitioned by whatever key the reduction groups on (run_id, job_type, source) — so under(:projections:<name>) is the cross-run family. See §Stage-record addressing for the role-major convention. The two addressings differ; the built one is run-major.
foxfire memoizes a completed run's folds as :projections: rows, written by rollup_* boundaries that fire on the run's terminal crossing. The wiring is data — config.yml triggers: match a record type_addr and fire a named boundary:
triggers:
- name: seal_task
when:
type_addr: ":events:task:complete"
then: rollup_task
- name: seal_run
when:
type_addr: ":events:run:completed"
then: rollup_run # first — seals :projections:<run>:run
- name: seal_reconcile
when:
type_addr: ":events:run:completed"
then: rollup_reconcile # seals :projections:<run>:reconcile
# seal_cost / seal_otlp / seal_envelope also fire on run:completed, in order
rollup_run runs first on run:completed; the cost, reconcile, OTLP, and envelope seals read the run projection it wrote. Each rollup_* boundary is a [B] reducer that ends in one Entity.upsert under :projections:<run>:<kind>. This is freshness: seal as built: compute once at the terminal crossing, store, serve forever. There is no watermark field and no declarative freshness: — the trigger table is the lifecycle.
Reads resolve off the sealed rows unioned with a live compose for the in-flight window. composed_runs lists the sealed run projections (list by type :projections:run) and unions the runs that have a :events:run:started event but no seal yet (live_run). composed_tasks unions sealed per-task grains with the tasks still in flight (live_tasks). A run therefore appears the moment it starts, and its sealed grain replaces the live one on the same hash once it completes.
The reduce is the whole story. It resolves to one of two realizations:
| Form | Realization | Built as | Notes |
|---|---|---|---|
| metric / set algebra | [N] (SQL) / [R] (Ruby) | the list / aggregate capabilities |
property-tested [N] == [R]; the OOM read paths |
| judgment | [B] (Ruby only) | the verify_run boundary and the rollup_* seals |
joins ledgers + reasons; never expressed in SQL |
Data-driven wiring (which projections exist, when they fire, where they land) with code-defined reduction for the hard cases — the same split as actor type: dispatch everywhere in the framework.
[target] The declarative engine would express the metric side as a closed reduce: verb vocabulary in the spec, each verb compiling to a native SQL template and a Ruby reference reducer:
| Verb | Meaning | [N] Postgres | [R] Ruby | Built? |
|---|---|---|---|---|
group_by: <field> |
partition rows before aggregating | GROUP BY |
group_by |
yes — aggregate |
sum: <field> / count: |
folds | native aggregates | sum / length |
yes — aggregate |
project: [<fields>] |
select a subset of keys | column projection | slice |
yes — list |
page: { after:, limit: } |
keyset window | keyset WHERE > cursor LIMIT n |
drop_while + take |
yes — list |
p95: [<fields>] |
exact 95th percentile per field | percentile_cont(0.95) |
nearest-rank | no |
latest_by: <field> |
last-writer-wins on a field | ORDER BY … DESC LIMIT 1 |
max_by |
no |
min: / max: |
folds | native aggregates | min / max |
no |
p95 today is nearest-rank in Ruby (envelope_rollup.p95); no Postgres percentile_cont path exists. latest_by is realized instead by the pointer/label mechanism (§Stage-record addressing).
[target] The declarative engine chooses a lifecycle by one property of the aggregate it folds: is the input closed (will never change) or open (still growing)? (Vocabulary borrowed from Marten's inline/async/live; landed on stock Postgres per the advaita-projection-engine decisions.) The built system realizes two of these already, without the declarative field:
freshness |
Aggregate | Compute trigger | Built as |
|---|---|---|---|
on_write |
cheap, always-current | every append | the pointer/label write at promotion (§Stage-record addressing) |
live |
any | every read | the list / aggregate capability, computed each call |
seal |
closed | first read after the aggregate closes | foxfire's rollup_* triggers on the terminal crossing |
incremental |
open | first read after the tip advances | not built — the envelope is a live TTL-cached scan, no watermark |
The seal lifecycle is the essential one: a completed run's trace is frozen, so its rollup — a metric document or a whole judgment — is computed once at close and served forever. Exact, zero recompute. The incremental lifecycle (fold new sealed rollups, advance a watermark) is target: the cross-run envelope is a full-store scan with a 60-second cache, not an incremental merge.
A built :projections: row is a plain wanderland-core-storage-engine record written by Entity.upsert — the payload is the fold, signed-from stays, and there is no watermark:
to_addr: ":projections:echo-20260717144738-d94b94:run"
from_addr: "boundary:rollup_run"
type_addr: ":projections:run"
payload: { run_id, run_name, status, job_type, task_count, compute: { credits }, processes: { … } }
at: "2026-07-24T17:15:00Z"
sig: "…" # signed-from stays
Invalidation is structural: only a terminal crossing writes the row, and a completed run's inputs are frozen, so the row is permanent. A re-seal (/admin/reseal) overwrites it through the same upsert.
[target] The declarative engine carries a watermark (the input tip the fold was computed at) and a realized: stamp in the payload, so a read can compare the watermark against the current tip and recompute an open aggregate. The built seal rows carry neither — closed aggregates never need the comparison.
The reduction's [N]/[R] realization resolves through the storage capability surface (case task-7d58f481) — crossings' capability addressing brought into the little sister, minus the walk and planner:
resolution: (address | type_addr, capability_name) --longest match--> mounted driver
realization: driver.capability?(name) → yes: driver.cap_<name>(addr, **args) no: Capabilities.<name>(driver, addr, **args)
Storage::Registry.capability(addr, name, **args) resolves the driver by longest-prefix match, then dispatches native when the driver advertises the capability, else to the shared in-memory Storage::Capabilities (the [R] reference). Three capabilities ship:
| Capability | Reads | Args | Fold |
|---|---|---|---|
list |
overlaid records under an address | project / where / type / page: { after, limit } |
project keys, filter, keyset-paginate; never hydrates unselected IOU leaves |
aggregate |
overlaid records under an address | group_by / count / sum / type / where |
one row per group, count and summed dot-paths, folded in address order |
pending |
the raw append log under an address | type / exclude |
the store-as-queue anti-join — one survivor per cell, latest payload, oldest first-seen first |
list and aggregate reduce over overlay_all (antiparticles + $merge applied), so they read current state. pending reads the raw log, because completion is a sibling lifecycle record at the cell, not an antiparticle. Native cap_list / cap_aggregate / cap_pending exist on both the Postgres and sqlite drivers.
The contract is named ops only, resolved at an address, realized native-or-fallback. It is not a query language, not crossings' compile→plan→execute→bridge pipeline, not the forking walk. See advaita-projection-engine for the scope boundary.
:projections:<run>:run — the per-run seal [B, built]The exemplar closed aggregate. rollup_run computes it once when the run seals and stores it forever. Values from the live echo.fibfizz run echo-20260717144738-d94b94:
# trigger: :events:run:completed → rollup_run
# :projections:echo-20260717144738-d94b94:run → payload:
run_id: echo-20260717144738-d94b94 status: succeeded job_type: echo.fibfizz task_count: 2
compute: { credits: 0.0000654 }
processes:
FIBONACCI: { samples: 1, realtime_ms: { p95: 3454 }, peak_rss_bytes: { p95: 45100000 }, cpus: 1 }
FIZZBUZZ: { samples: 1, realtime_ms: { p95: 2460 }, peak_rss_bytes: { p95: 44900000 }, cpus: 1 }
The per-run p95 is nearest-rank over that run's task samples, not percentile_cont. [target] Under the declarative engine this is over: :events:task, partition: run_id, reduce: rollup_process, freshness: seal, addressed role-major at :projections:run_trace_rollup:<run_id>.
wisp's profile consumes a per-job_type stat block (advaita-nextflow-router-data-model Stage C). EnvelopeRollup builds it by walking every plan of the job_type, matching run names to runs, and pulling those runs' tasks — a full-store scan cached for 60 seconds. rollup_envelope also fires at seal and writes a per-run envelope contribution, but the read path is the live scan.
# GET envelope(job_type) → live rollup, 60s TTL cache
job_type: echo.fibfizz source: "foxfire · live rollup" n_runs: 15
processes:
FIBONACCI: { samples: 15, fanout_per_run: 1.0, cpus: 1, gpu: 0, realtime_ms: { mean: 3450, p95: 3458 }, peak_rss_bytes: { mean: 44951689, p95: 45113344 } }
FIZZBUZZ: { samples: 15, fanout_per_run: 1.0, cpus: 1, gpu: 0, realtime_ms: { mean: 2452, p95: 2475 }, peak_rss_bytes: { mean: 44700467, p95: 46157824 } }
The p95 is nearest-rank in Ruby. Fuzzy across runs, exact within each — accepted for sizing (Graeme, 2026-07-29). [target] freshness: incremental over sealed per-run rollups with a watermark removes the full-store scan; it is not built.
The judgment class Graeme flagged as "the kind of thing I need to materialize." Today it is two records, not one:
verify_run — seal-chain verification (links + signatures + a timeline check). Computed live on every read; it walks the whole :events:<run>: spine each call. Not seal-cached.:projections:<run>:reconcile — plan-vs-actual drift per process (queue hop, resource match, fit). Seal-cached by rollup_reconcile on run:completed; reconcile_read serves it directly and returns 404 when absent.# :projections:synth-20260724030635-6550fc:reconcile → payload:
run_id: synth-20260724030635-6550fc job_type: synth.old_lady
processes:
SWALLOW:
planned: { queue: workers-batch-cpu, flavor: spot, spot_eligible: true }
placed: { queue: workers-batch-cpu, flavor: on-demand, node: gf2a9c4 }
used: { peak_rss_mb: 45, realtime_ms: 8455 }
drift: { queue_hop: false, resources_match: true, fit: true }
[target] One run_verdict document — verdict, summary, findings, drift, retry absorption — materialized once at seal at :projections:run_verdict:<run_id>, folding verify_run and reconcile into one [B] boundary. Not built; the reconcile half is seal-cached, the verify half is still live.
acq_manifest — per-source label [built via the pointer]The DrugBank delta gate's provider. An acquisition run writes an immutable per-run instance record and moves a promoted label at it; the gate resolves the label (advaita-nextflow-router-data-model Touch points). The label is a Storage::Pointer — a :types:pointer record whose payload names the instance address, dereferenced through Pointer#deref (§Stage-record addressing).
# resolve :projections:acq_manifest:drugbank:promoted → instance
source: drugbank version: "001419158" status: promoted run_id: <nf run id>
image_digest: "sha256:37159dba…"
metrics: { tables: { drugs: 28151, products: 323875, clinical_trials: 246830, conditions: 141580 }, totals: { tables: 122 } }
First run for a source → no prior promoted label → resolves null cleanly (seeds the baseline). All band/threshold policy stays consumer-side; the provider only stores and serves.
The runs list once dragged every fat task trace into Ruby to render a count and cost column. It now resolves off the sealed run projections through the list capability, unioned with the live compose for in-flight runs:
# Schema.run_projections → list(":projections:", type: ":projections:run")
[ { run_id: …, run_name: echo-…, status: succeeded, job_type: echo.fibfizz,
started_at: …, duration_ms: 15509, retries: 0, task_count: 2, total_cost: 0.0000654 }, … ]
task_count and total_cost are sealed into the run projection at close, so the read projects stored columns and never hydrates a task trace. The aggregate capability serves the same need for any per-group count or sum without carrying payloads out.
list and aggregate fold over overlaid state (antiparticles applied), never the raw append log. pending reads the raw log by design — completion is a sibling record, not an antiparticle.{_iou, _size} markers unless selected. This is the OOM fix at the record level.:projections: rows are signed like every record; per-row seal-chain checks are verify_run's job, not the reduction's.$merge path; the prior fold remains a signed fact. A stale entity is corrected by refold, not deleted (a tombstone is a flag readers skip).list, aggregate, and pending.The data model is the spec; these keep a change honest:
capabilities_parity_spec, capabilities_aggregate_parity_spec, capabilities_pending_parity_spec. In-db and in-memory can never silently diverge.refold_run re-runs the fold. A projection row is a cache, never a source of truth (lebowski-corollary).live/native reduction is shape→shape → a scenario (operation: + expected:); a signed-ingest write is rspec (wanderland-testing-ai-primer).refold_run re-derives the run/task entities from the append-only event trail. It is needed when the fold's identity rules change under existing history: events keyed on the Nextflow session id braided several submissions into one address space (a resumed run adopts its ancestor's session). Each event payload carries its own run_name, so the projection is recoverable — group the braid's events by run_name, replay the fold per group under the true identity, and tombstone the stale entities. The raw trail stays at its original addresses; only the composed entities move.
[target] A refold as a traceable generation, not a tombstone-and-overwrite. The record schema would carry trace_addr — crossings' fourth address (the causal pointer) — so a refold writes a fresh full projection whose trace_addr points at the generation it supersedes, and walk(trace_addr) recovers gen N → … → gen 0. The name is trace_addr, not trace: foxfire already uses trace for the Nextflow resource-trace payload. Not built — the current refold_run tombstones the stale entity and rebuilds in place.
# [target] shape:
trace_addr: ":events:run:<run_id>@<prev-gen-at>" # the generation this one reprojects from
payload: { …, gen: 2, refolded_from: 1, reason: "session-braid split by run_name" }
A stage-written record carries two coordinates: a role (stable, cross-run — the pipeline/source and the step) and an instance (run_id). Address them as one family by putting a ref in the leaf slot, where a ref is either a concrete instance id or a moving label — exactly Docker's repository:tag:
:projections:acq_manifest:<source>:<ref> # <ref> ∈ { <run_id> | <version> | latest | promoted | candidate | … }
| Docker | here | mutability |
|---|---|---|
repo (library/postgres) |
the role address :projections:acq_manifest:<source> |
— |
@sha256:… digest |
instance ref :…:<run_id> (and the export :…:<version>) |
immutable — always present, pins a run |
:tag (latest, 16.2) |
label ref :…:latest, :…:promoted |
moving — overlay, last-writer-wins |
| moving a tag | write a new label record (overlays prior) | append-only |
| tag history | walk the label's prior records | the promotion history |
A label is a thin pointer — the built Storage::Pointer. It is a :types:pointer record whose payload is { points_at: ":…:<run_id>", path: <dot-path>? }; Pointer#deref walks pointer → pointer → source (cross-mount through the store's own routing), applies each hop's path, and hydrates IOU leaves once on the final value. So pinning a specific run and asking for current coexist, and several tags can name one run.
Status becomes a tag. The candidate→promoted lifecycle is two labels: an acquisition run writes its instance and moves :…:candidate; promotion moves :…:promoted (≡ :…:latest). The "compare against the last promoted, not the last emitted" subtlety dissolves — the gate reads the promoted label, and a failed candidate never moved it.
The gate's reads:
latestManifest(source) = resolve :projections:acq_manifest:<source>:promoted → instance. One overlay, O(1).manifests(source, last: N) = walk the promoted label's prior records back N.:…:<run_id> or :…:<version> (immutable, always present).This is the built latest_by — the label is written on promotion, so a scan-reduce over the family is only the rederivation oracle (the [R] reference), never the read path.
[target] The role-major convention above is the target addressing for one-per-run records (manifest, run_verdict, plan), so under(:…:<role>) is the cross-run family. foxfire's built projections are addressed run-major instead (:projections:<run>:<kind>), so their cross-run cut is a type scan. The pointer/label mechanism is built; the role-major projection addressing is not the shape foxfire uses today.