wanderland-core-record-dataflow

Frontmatter

diataxis: Explanation + Reference
style: ASD-STE100
status: design of record. The write and read verbs carry the store terminals today. The parse, validate, transform, and format layers are the next slices.
case: task-9a2387b4-4349-4cff-924e-d66d5886cdb2
parent_case: task-ffa7a0d3-691d-4b02-8fd4-3270bbf35abc
tickets: ADV-281
feature: one read and one write verb for every record, driven by the record's type

Legend

• waist — the single canonical shape every format converges to. One shape in the middle, so many formats reach many consumers through it.
• frozen — the waist shape never varies per format. Every format-specific rule lives in a codec at an edge, never in the waist.
• IR — the intermediate representation. The waist made concrete: the record payload, always structured data.
• structured data — an abstract syntax tree, or a keyed map. Never an opaque string. Structure is first-class, so a program can walk it, match it, and change it.
• carrier — the record as it rides one arc, and gains a value at each hop.
• hop — one transform on an arc.
• layer — a mirrored pair of hops. One rides the write arc, one rides the read arc, and both key on one pivot.
• pivot — the value a layer keys on. The MIME, the type schema, the graph holes, or the address.
• type record — the one record that declares a record kind. It holds the schema, the write policy, the codec, and the triggers. Address :type:<name>. Every rule about how a record is written, read, viewed, and acted on lives in this one record. A read resolves it on every operation, so it is the single switch for all data behaviour.
• codec — the parsers and views for a type. A parser converts formatted input to IR by MIME. A view converts IR to formatted output by MIME. The codec is declared on the type record, in a codec block beside the schema, the write policy, and the triggers. One type record holds both how a reader views a record and what the record does. A flat :codec:<leaf> address is a transitional fallback for a type that declares no codec block.
• parse — a hop: formatted input to IR, through the codec parser.
• format — a hop: IR to formatted output, through the codec view. The inverse of parse.
• validate — a hop: check the IR against the type schema.
• transform — a hop: project the IR into the shape a view needs. The read-side mirror of validate.
• diff — a hop: the desired IR against the composed current, to a delta.
• link — a hop: fill the IR holes from the child records, to the whole. The inverse of diff.
• append — a hop: a delta to one appended record. The write terminal.
• overlay — a hop: the append log at an address to the composed IR. The read terminal.
• hole — a node in a stream that names a child record. A plain hole is { type: loom:hole, ref: <addr> }. A frontmatter hole adds kind: vfence. Link fills it on read.
• vfence — a typed child record that holds a keyed map, such as a frontmatter block. It has its own type and schema. A hole in the parent points at it, and a poke sets one field.

The one idea

Delayed compilation over a frozen narrow waist, stored as a content-addressable graph. A write compiles formatted input into one canonical shape and stores it. A read walks the graph and recasts that shape into whatever format the reader asks for. The store holds the waist, never a formatted blob.

The waist is always structured data — an abstract syntax tree, or a keyed map. Markdown compiles to a token tree, Ruby source to a syntax tree, and a form to a keyed map. Each one is JSON, so one store shape holds every kind. One reader then recasts a stored tree into a second format — a token tree to HTML, a syntax tree to code or to a graph. The structure survives the round trip, so a program can act on it.

The narrow waist earns its keep by arithmetic. With N input formats and M output formats, a waist needs N parsers and M views. That is N plus M adapters, against N times M direct interfaces. The saving holds only while the waist stays frozen. A format-specific field in the waist re-couples the ends, and the count returns to N times M. So the rule is strict: the waist carries pure structure and content, and every format rule lives in a codec at an edge.

A new record type is then a declaration. State the type, its schema, how it reads, and how it writes. The verbs and the chains stay fixed, because the waist never moves.

The frozen narrow waist

Structure matters as much as content, because a keyed map and a token tree are both walkable. One primitive then reads any record: !Pluck digs a path, ShapeMatcher matches a shape, and the diff compares two trees. None of them knows the record kind. The waist forbids an opaque string for this reason. A blob is content with no structure, so no primitive can act on it, and the waist stops being programmable.

The waist is the pivot of the whole model. Everything to its left is a write that compiles toward the canonical shape. Everything to its right is a read that recasts from it. The store sits exactly in the middle, and the arcs mirror around it.

A child record is not a special field. A frontmatter block, a comment, a section — each is a hole in the parent stream, and link fills it on read. So the structure is uniform: one stream of nodes and holes, and one link that fills them.

The same skeleton serves every record kind: an optional header, a metadata vfence, and line children. A ticket or a task carries a header. A comment does not, because its metadata is its identity. Notebook tasks already run this skeleton, so a new type is a codec over one proven shape.

The flow

Green marks a hop already built and wired. Amber marks the next slice. The codec and the schema each feed both arcs, which is the symmetry.

direction: down
classes: {
  prod: { style.fill: "#e4efe0" }
  dev:  { style.fill: "#f4ecd4" }
  doc:  { shape: document; style.fill: "#edeef1"; style.font-size: 20 }
}

codec: "Codec (:codec:<type>)" { class: doc }
schema: "Type schema" { class: doc }

parse: "parse — formatted → IR" { class: dev }
validate: "validate — conform to schema" { class: dev }
diff: "diff — IR → delta" { class: prod }
append: "append — delta → log" { class: prod }

ir: "Frozen waist — the JSON IR at an address" { shape: cylinder }

overlay: "overlay — log → IR" { class: prod }
link_hop: "link — fill holes → whole" { class: dev }
transform: "transform — project to the view" { class: dev }
format: "format — IR → formatted" { class: dev }

parse -> validate: emits the IR
validate -> diff: "passes [schema ok]"
diff -> append: emits the delta
append -> ir: writes the log
ir -> overlay: composes the log
overlay -> link_hop: emits the IR
link_hop -> transform: fills the holes
transform -> format: projects the IR

codec -> parse: feeds the parser
codec -> format: feeds the view
schema -> validate: gates the shape
schema -> transform: shapes the target

The symmetry — four layers, four pivots

Each write hop has a read hop as its inverse, and each pair keys on one pivot. Only the store layer is always present. The other three layers default to a pass-through when a type declares nothing.

Layer	Write hop (in)	Read hop (out)	Pivot	Default when the type declares nothing
codec	parse (formatted → IR)	format (IR → formatted)	the MIME	Parse is identity for a JSON body. Format returns the raw payload.
shape	validate (conform to schema)	transform (project to the view)	the type schema	Both are off. The IR passes through.
graph	diff (IR → delta)	link (fill holes → whole)	the holes	Diff is a JSON sparse diff. Link fills any holes. No holes means identity.
store	append (delta → log)	overlay (log → IR)	the address	Always present. Append signs the record, and overlay verifies it.

The shape layer holds the two optional slots, and they are independent. A type may transform with or without a validating schema. It may validate with or without a downstream transform. Validate fits the incoming toward the canonical schema. Transform fits the canonical toward the outgoing view. Same pivot, mirrored direction.

Integrity is not a fifth layer. append signs the record, and overlay verifies it. So the trust check rides the store terminals.

The write terminal — the triplet

The store layer's write hop is one boundary the op selects, not a fixed append. The type's write: field picks one of four, and each operates on the IR alone, so the terminal never reads the record's meaning:

• append — write the whole IR as one record, with no diff. Each write is a new record, so this is the event case: an audit entry, a signed collector fact, a log line.
• merge — a sparse diff of the IR against the composed current, appended as a $merge delta of the changed keys. An absent key stays untouched, so a shared address is safe for many writers.
• replace — a full-state reconcile. A $snip` carries each removed or changed key's old value, then a `$set writes the new, so a dropped key retracts. Use it on a single-owner address.
• create — write the whole IR as one record, only at an empty address. The driver checks the address and writes in one step, so the first writer wins and each later write gets a 409. A chain whose next address comes from its tail, such as a lease chain, uses it, so the chain has one next link.

So the diff is internal to merge and replace, and append skips it — the graph layer's diff is governed by the write policy, not always present. All four end in the append-only log; they differ only in what they append. This folds the two write paths into one: the signed collector gate's emit is append under this model, so collect and poke become the same funnel with a different policy.

The Type Record

A type record declares one record kind in one payload. It carries the schema (how an instance validates), the write policy (append, merge, or replace), the codec (the parsers and views), and the cache (the projection and its watermark). One read of this record answers how a record is written, validated, viewed, and rolled up. The example is :type:run, a projected run.

to_addr: ":type:run"          # instances carry this as their type_addr
type_addr: ":type:type"       # every type record is typed by the terminus meta-type
payload:
  role: document              # container | item | document | attachment | trait | machinery
  write: replace              # the write terminal — append | merge | replace

  schema:                     # how an instance validates
    type: object
    properties: { events: { type: array } }
    required: [events]

  cache:                      # the projection and its watermark
    at: "{run}:projection"    # memoize the rollup here
    "on": ":signals:run:complete"  # the terminal signal fires the projection eagerly
    key: run
    render: text/markdown

  codec:                      # the parsers and views, inline
    parsers:
      text/markdown: { via: boundary, ref: loom_ingest }
    views:
      text/markdown: { via: template, template: "# Run <%= data['run_name'] %> ..." }

Every type record is typed :type:type, the terminus. The schema allows extra keys, so the codec and cache blocks ride on the one payload. Codec.resolve reads payload.codec first. A separate :codec:<leaf> record is the transitional fallback for a type that has not moved its codec inline. cache.on is the terminal signal that fires the projection, the shipped watermark. A broader triggers field is the next slice.

The write arc — a markdown ticket

The carrier is a jira ticket. It arrives as markdown: a frontmatter block, and a body.

• parse — the parser decomposes the text/markdown ticket into a set of records, through :codec:jira-ticket. The root holds the heading, and a stream of body nodes and holes. The frontmatter is a child vfence, and a hole names it. Each comment is its own record, and a hole names it. This is the loom write_records shape.
• validate — the check confirms each record against its type schema. The ticket carries a schema, so this slot runs.
• diff — the diff compares each record against its composed current. A jira sync has one owner, so it runs a full-state reconcile, and a dropped field retracts.
• append — the append writes each record's delta at its own address, from :sync:jira@<as_of>. So one parse produces many records, each appended on its own.

# the input the write verb receives — the ticket as markdown
to_addr: ":jira:PROJ-812"
type_addr: ":type:jira-ticket"
payload: |
  ---
  status: In Progress
  assignee: gfawcett
  priority: High
  ---
  # Login fails on empty password

  The form posts an empty body when the field is blank.

# parse yields a set of records; each is appended at its own address.
# the ticket root — the heading sits beside the stream, the holes sit in it
to_addr: ":jira:PROJ-812"
type_addr: ":type:jira-ticket"
payload:
  heading:
    depth: 1
    text: Login fails on empty password
  stream:
    - type: loom:hole
      ref: ":jira:PROJ-812:vfence:frontmatter"
      kind: vfence
    - type: paragraph
      children:
        - type: text
          value: The form posts an empty body when the field is blank.
    - type: loom:hole
      ref: ":jira:PROJ-812:comment:1"

# the frontmatter — a child vfence, so a poke sets one field
to_addr: ":jira:PROJ-812:vfence:frontmatter"
type_addr: ":type:jira-ticket-frontmatter"
payload:
  kind: frontmatter
  data:
    status: In Progress
    assignee: gfawcett
    priority: High

The read arc — the same ticket back out

• overlay — :jira:<KEY> composes to the root IR. The stream carries holes, one for the frontmatter vfence and one per comment.
• link — link fills each hole with a peek of the child. The frontmatter vfence and every comment resolve, recursively.
• transform — transform projects the IR for the requested view. The full page keeps every field, the list item a few.
• format — the projected IR renders through :codec:jira-ticket views. text/markdown+full renders the page, text/markdown+listitem a row, and text/html recasts the same IR to HTML.

# what overlay returns — the root IR, with holes where the children sit
to_addr: ":jira:PROJ-812"
type_addr: ":type:jira-ticket"
payload:
  heading:
    depth: 1
    text: Login fails on empty password
  stream:
    - type: loom:hole
      ref: ":jira:PROJ-812:vfence:frontmatter"
      kind: vfence
    - type: paragraph
      children:
        - type: text
          value: The form posts an empty body when the field is blank.
    - type: loom:hole
      ref: ":jira:PROJ-812:comment:1"

# link fills the vfence hole with this
to_addr: ":jira:PROJ-812:vfence:frontmatter"
type_addr: ":type:jira-ticket-frontmatter"
payload:
  kind: frontmatter
  data:
    status: In Progress
    assignee: gfawcett
    priority: High
---
# and the comment hole with this — no heading; its own vfence hole leads its stream
to_addr: ":jira:PROJ-812:comment:1"
type_addr: ":type:jira-comment"
payload:
  stream:
    - type: loom:hole
      ref: ":jira:PROJ-812:comment:1:vfence:frontmatter"
      kind: vfence
    - type: paragraph
      children:
        - type: text
          value: Reproduced on staging.
---
# link resolves that comment's vfence hole recursively — the view formats "author · date" from this
to_addr: ":jira:PROJ-812:comment:1:vfence:frontmatter"
type_addr: ":type:jira-comment-frontmatter"
payload:
  kind: frontmatter
  data:
    author: dsmith
    created: "2026-09-20"
    id: 1

The round trip is the balance check: markdown to IR to markdown is identity, and markdown to IR to HTML is a recast. Same waist, many formats.

Runtime sequence

shape: sequence_diagram
adapter: Adapter
poke: poke verb
peek: peek verb
codec: "Codec (:codec:jira-ticket)"
store: Record store

adapter -> poke: "write (text/markdown)"
poke -> codec: parse the body
codec -> poke: the mdast IR
poke -> poke: validate, then diff
poke -> store: append the delta
adapter -> peek: "read (text/markdown+full)"
peek -> store: overlay the address
store -> peek: the IR with holes
peek -> peek: link, then transform
peek -> codec: format the IR
codec -> peek: the rendered page
peek -> adapter: the response

What already holds

Most of the funnel exists. The build wires these behind the two verbs and the codec, and adds no new machinery.

• parse :: the loom write_records set (root, child vfences, section records), or identity for a JSON body. Built, not yet a codec parser.
• validate :: not built.
• diff :: Storage::Entity.upsert, sparse or full. Built, wired in poke.
• append :: driver.append, the write terminal. Built, wired in poke.
• overlay :: driver.overlay, antiparticle and operator applied. Built, wired in peek.
• link :: the loom peek recursive hole fill. Built in loom, not yet the core read verb.
• transform :: the format-by-type select and filter. Designed, not built.
• format :: type_template_formatter, the template renderer. Built, unwired.

The format-by-type work is .

Open items

• The read-side pair for validate is transform, so the arcs balance four and four. A separate signature verify is not a layer. It rides the store terminals.
• The type record carries the codec inline — a codec block holding parsers and views, each keyed by MIME, beside the schema, the write policy, and the triggers. A parser names a boundary the way a view names a renderer, the same { via, ref } shape both directions. Resolution reads the codec off the type record. A :codec:<leaf> address is a transitional fallback for a type that declares no codec block.
• The hole and link contract mirrors the pointer deref. A hole names a child address. Link is a recursive peek, cycle-safe. See wanderland-core-storage-data-model#the-pointer-the-proposed-data-model.