Make the datastore the owner of truth. The move is to lift the fold down into the store — as materialized projections computed once and read cheaply — so foxfire trends toward a stateless reader. Two layers of that ship today: a named-capability read surface in wanderland-core, and a set of seal-time rollup projections foxfire memoizes when a run closes. The generic, data-declared projection engine that would unify them is the target this node also describes.
The shapes this engine consumes are the contract: advaita-projection-data-model (the correctness gauge — read it alongside). This node is the why and the how; that node is the what.
Storage::Registry.capability dispatches a named read-reduction to the mounted driver's native cap_<name> or the in-memory Storage::Capabilities fallback. Three capabilities: list, aggregate, pending. The fallback is the reference semantics; parity specs assert native equals fallback. This is the layer that stops a read from dragging fat task traces into Ruby to render a column.config.yml triggers: fire rollup_* boundaries on a run's terminal crossing, writing :projections:<run>:<kind> rows. Reads resolve off those sealed rows unioned with a live compose for the in-flight window. reconcile is seal-cached; the run list, run detail, and per-task grains read the seals.Storage::Pointer dereferences a :types:pointer record to its target (hydrate-last, cycle-guarded) — the Docker-convention promoted/latest labels.reduce: vocabulary with exact p95, a watermark, /reload/projections, and refold-as-generation. Sections marked [target] describe it. It is not built; foxfire's trace_otlp marks itself a stopgap until it lands (cases task-7d58f481 and task-06fe434c).replicas: 1 remain. See §De-singularize.crossings (big sister) four-address store · three-way FORKING WALK ·
[crossings-architecture](crossings-architecture) capability → compile → plan → execute → bridges ·
Pause/Fetch/Splice · the store IS the queue
│ keep the record shape + capability ADDRESSING
│ DROP the walk + planner + bridges
▼
wanderland-core storage engine (little sister) append-and-read-by-prefix · overlay via
[wanderland-core-storage-engine](wanderland-core-storage-engine) antiparticles · signed-from · IOU blobs
│ bring capability addressing BACK, scoped to storage reductions
│ swap compile-to-bridge-plan → dispatch-to-driver-native-or-fallback
▼
storage capability surface (task-7d58f481) BUILT: list / aggregate / pending AT an address ·
resolved longest-match · [N] native / [R] fallback ·
fallback IS reference semantics
│ memoize the fold · address the result · defer recompute
▼
seal-cached projections (foxfire, built) config triggers → rollup_* → :projections:<run>:<kind> ·
read as sealed ∪ live
│ [target] generalize: data-declared spec · watermark · /reload
▼
the declarative projection engine (target) closed/open + watermark · seal-cached materialization ·
/reload · full de-singularize
The big sister (crossings-architecture) is the full capability engine: a request (address, capability, request_context) triggers a reductive three-way walk that compiles a plan, previewed then executed through bridges — its four phases are the fundamental algorithm (Pause → Fetch → Splice → Continue). The little sister kept the record shape and dropped the walk and planner. This engine reintroduces exactly one idea from the big sister — capabilities addressed at a type_addr — scoped to storage reductions, and realizes them by dispatch to the mounted driver, not by compiling to bridges. It does not rebuild the planner.
A projection is a memoized fold over the append-only ledger, and reading it is Pause → Fetch → Splice → Continue:
:projections: row.:projections:<run>:<kind> row written at the run's terminal crossing.watermark.live_run / live_tasks) and unioned with the sealed rows, so it appears the moment it starts.Burn the CPU iff you need it is deferring the Splice until a query forces it — delayed binding, the same move as lazy evaluation everywhere.
| Layer | Owns | Built form |
|---|---|---|
| Storage engine (wanderland-core-storage-engine) | the append log + overlaid current state | append-only, signed, prefix-mounted |
| Capability surface (task-7d58f481) | compute a reduction once, portably | list / aggregate / pending, [N] native or [R] fallback (reference) |
| Projection / cache | when NOT to recompute — materialize + memoize | foxfire's config-triggered rollup_* seals writing :projections:<run>:<kind> |
The capability answers "get me the count and cost without OOMing" ([N] GROUP BY); the projection answers "don't run even that again for a sealed run." They compose: capability = compute the fold; projection = memoize the fold. [N]-vs-[R] and cache-vs-source are both the same fold relocated to a cheaper coordinate, never new truth. refold_run proves rederivation. [target] a single declarative engine that both computes and memoizes from one spec is the layer these two grow into.
Crossings' capability addressing, minus the walk — built:
resolution: (address | type_addr, capability_name) --longest match--> mounted driver
realization: driver.capability?(name) → yes: driver.cap_<name>(addr, **args) no: Capabilities.<name>(driver, addr, **args)
Storage::Registry.capability(addr, name, **args) resolves the driver by longest-prefix match, then dispatches native or fallback. The three built capabilities are list (project / where / type / keyset page), aggregate (group_by / count / sum), and pending (the store-as-queue anti-join).[N] (the driver's cap_<name>, compiled for its backend) and [R] (Storage::Capabilities, a generic scan-over-overlay + Ruby reduce). Both the Postgres and sqlite drivers advertise all three natively via capability?.capabilities_parity_spec and the aggregate/pending parity specs) so in-db and in-memory can never silently diverge.What it is not: no compiler, no plan/preview/execute pipeline, no bridges, no forking walk, no query language. Named ops only.
foxfire memoizes a completed run's folds through data-declared triggers. config.yml triggers: match a record type_addr and fire a named rollup_* boundary; rollup_run runs first on run:completed, then cost, reconcile, OTLP, and envelope read the run projection it wrote. Each rollup_* is a [B] reducer ending in one Entity.upsert under :projections:<run>:<kind>. This realizes freshness: seal without a declarative field: the trigger table is the lifecycle, and a completed run's frozen inputs make the row permanent.
The reduce split holds at this layer. A metric read uses the list / aggregate capability (data-driven, native-or-fallback); a judgment is a named Ruby boundary (verify_run, the rollup_* seals). Data-driven wiring with code-defined reduction for the hard cases — the same split as actor type: dispatch across the framework. Forcing a judgment into PL/pgSQL would re-grow the crossings planner the little sister dropped.
[target] The declarative engine generalizes the wiring: projections declared as data (over / partition / reduce / into / freshness / seal_when — advaita-projection-data-model §spec), a metric reduce: verb compiling to an SQL template plus a Ruby reference reducer, and one materialization scheduler deciding when (on write / on read / at seal / on cron) and where to persist. foxfire's per-run seals are the seal case of this, hand-wired.
The record Graeme surfaced — verdict: clean, summary, findings, per-process planned-vs-placed-vs-used drift, "1 death absorbed by retry: SWALLOW" — is the proof the model generalizes past metrics. Today it is two records, not one:
verify_run joins the run's whole signed spine and checks the seal chain (links + signatures + a timeline judgment). Computed live on every read — it walks :events:<run>: each call. Not seal-cached.:projections:<run>:reconcile is the plan-vs-actual drift per process. Seal-cached by rollup_reconcile on run:completed; reconcile_read serves the stored document and 404s when absent.[target] One run_verdict [B] boundary, freshness: seal, folding verify_run and reconcile into a single document at :projections:run_verdict:<run_id>, served forever. Not built: reconcile is sealed, the seal-chain verify is still live. (Worked values: advaita-projection-data-model §3.)
/reload/projections[target] Push a ConfigMap, hit an endpoint, projections update in-process. Two halves of different weight, neither built:
REFRESH … CONCURRENTLY needs a unique index; treat destructive DDL as a migration; never take a lock that stalls ingest). Additive by default.Today a projection change is a code change to a rollup_* boundary and a config.yml trigger, applied by redeploy.
foxfire is replicas: 1 because part of the read model still lives in one Ruby process. The seq/seal-chain half of the ceiling is gone; the read-cache half remains.
:events:event:<run_id>); seq is a read-time ordinal, never written. The imperative seq = length; append that needed a mutex is gone. The @write_mutex now guards only reap_run's read-status-then-seal, so concurrent reaper finalizers cannot double-seal a run.@read_cache / @generation) that serves the hot reads and invalidates on write; it is correct only because one process is the single writer. A second writer process would invalidate it. Moving the fold to trigger-maintained entities or on_write projections, and letting MVCC give read consistency, retires the cache and lifts the single-writer ceiling.Target: foxfire as a stateless facade over store-owned projections — replicas: N. The GraphQL schema (schema.rb) already resolves :projections: rows as fields — the run list, run detail, and reconcile read the seals — so widening replicas turns on retiring the generation cache, not on rewriting the read surface.
pg_cron + trigger tables. No TimescaleDB, no pg_ivm for load-bearing paths. pg_ivm maintains only count/sum/avg/min/max (no percentiles) and is flagged not-prod-ready; Timescale continuous-aggregates + tdigest/uddsketch were the powerful alternative (real-time materialized+tail union is exactly the closed/open+watermark model, and the sketches give cheap incremental p95) — declined for dependency weight and sqlite parity. Local/sqlite runs the [R] fallback everywhere; parity stays 1:1.percentile_cont for exact per-run p95 at seal, the cross-run envelope merging exact per-run p95s (fuzzy across runs, exact within). Today p95 is nearest-rank in Ruby (envelope_rollup.p95); no percentile_cont path exists yet.Ecosystem reference: Marten (.NET/Postgres) is the projection-lifecycle vocabulary borrowed here (inline/async/live + rebuild-from-events = our refold). pg_ivm = incremental matviews, aggregate-limited. TimescaleDB continuous aggregates + uddsketch/tdigest = closest match to closed/open+watermark+cheap-p95, declined. Nobody ships "YAML spec → PL/pgSQL-or-Ruby, hot-reload, sqlite fallback" — that compiler is the bespoke work over standard parts.
Storage::Capabilities fallback + Registry.capability dispatch + list / aggregate / pending + native cap_* on both drivers + parity specs. foxfire's run list and detail read through it. This killed the OOM path — the read no longer drags fat task traces into Ruby to render a status column.rollup_run / rollup_task / rollup_reconcile / rollup_cost / rollup_otlp / rollup_envelope, wired to the terminal crossings, writing :projections:<run>:<kind>. Reads resolve sealed ∪ live.Storage::Pointer and the Docker-convention labels.resume_state on :events:run — projected per-step results keyed by step so a drone rehydrates its --state from foxfire (case task-f84ebcfa).watermark, freshness lifecycles, /reload/projections, the exact-p95 native verb, and the composite run_verdict document. This is what the trace_otlp stopgap waits on.replicas: N.The general engine must subsume foxfire's config-triggered rollup_* seals without duplicating them and without deepening the single-writer ceiling. These constraints hold for the build.
rollup_run writes :projections:<run>:run first, then rollup_cost / rollup_reconcile / rollup_otlp / rollup_envelope read it on the same :events:run:completed crossing. The engine must express "fire B after A on one crossing, where B's over is A's into" — projection-over-projection with intra-crossing order. An independent or async trigger model breaks the chain.@read_cache / @generation (correct only at replicas: 1) and the stored watermark are the same freshness concern at two layers. The engine's watermark replaces the cache. It does not run alongside it, and engine writes stay concurrency-safe without the @write_mutex.seal projection read returns the sealed row for a closed partition and a live compute for an open one. foxfire's composed_runs / composed_tasks union is that pattern hand-rolled. The engine owns it, so a consumer never rebuilds the union.:projections:<run>:<kind>, so "every run of a kind" is a type scan). The role-major target (:projections:<name>:<partition>, so under() is the cross-run family) is a different shape. The into: template emits either. Pick one standard and align foxfire before a projection reads across projections.envelope_rollup.p95); exact percentile_cont is the target. The reference reducer must compute exact percentile too, or p95 carries a documented tolerance — else the parity test fails by construction.rollup_* boundary returns a value, and the engine applies into, the upsert, and signed-from. The built rollups embed their own Entity.upsert and must split to generalize.trigger / freshness / into / watermark runs over Ruby reducers, which the built rollups already are. Native-SQL matview reducers add DDL reconciliation (REFRESH … CONCURRENTLY, a unique index, no ingest-blocking lock) and stay deferred until the declaration model is proven.Wanderland::Projection already names the chain: sub-step reach-back marker. The engine takes a different name (for example Storage::Materialization).pending stays a read capability :: it reads the raw log because completion is a sibling record, is inherently live, and takes no freshness lifecycle.verify_run walks :events:<run>: only). Refold-as-generation's trace_addr stays distinct from the seal-chain's trace.The engine is general when all six rollup_* seals, trace_otlp, and the promoted manifest label are declarations, with nothing hand-rolled left in foxfire. The hard cases are rollup_envelope (a cross-run merge — incremental plus a watermark) and the ordered rollup_run-then-dependents chain.
overlay_sql vs a materialized projection./reload DDL reconciliation safety — additive-only vs migration-gated destructive changes.refold_run is built as a re-derivation: it groups a braided run's events by run_name, replays the fold per group under the true identity, and tombstones the stale entity (advaita-projection-data-model §Generation lineage).
[target] Refold as a new generation that traces to the one it supersedes, reintroducing crossings' fourth address, trace_addr (the causal pointer). A refold would write a fresh full projection whose trace_addr points at the prior generation; reads take the head, and walk(trace_addr) recovers gen N → … → gen 0. The name is trace_addr, not trace — foxfire already uses trace for the Nextflow resource-trace payload. Not built: the current refold tombstones and rebuilds in place.
The run-free "current" reads (latestManifest, the promotion pointer) are the built Storage::Pointer: a pointer is an instance record whose leaf ref is a label — :projections:acq_manifest:<source>:promoted / :latest, exactly Docker's repo:tag (run_id ≈ immutable digest, label ≈ moving tag). Pointer#deref walks the chain unhydrated, applies each hop's path, and hydrates the final value once. latestManifest = resolve the promoted label; manifests(last: N) = walk it back N. Status becomes a tag (candidate → promoted are labels), so "compare against the last promoted, not the last emitted" is structural, not a filter. The label is written on promotion, so latest_by scan-reduce is only the rederivation oracle. Full role/instance/label convention: advaita-projection-data-model §Stage-record addressing.