Wanderland

Wanderland Core: Storage IOU Drivers

The Pattern

Every write to storage produces a crossing in the central SQLite index. The crossing always lives at its to_addr. The payload is either inline (small data) or an IOU marker pointing at where the content actually lives.

SQLite (always)
  to_addr:    :pki:keys:my-key
  from_addr:  :system:pki
  type_addr:  :types:pki:key
  sig:        "abc123..."
  at:         "2026-04-10T12:00:00Z"
  payload:    { _iou: "uuid", _driver: "file_store", _args: { path: "/keys/my-key.pem" }, _size: 8192 }

The SQLite index is the source of truth for what happened, when, and signed by whom. Content storage is just where the bytes live. You can swap content backends without losing provenance. You can query across all writes without hitting every backend.

IOU Marker Shape

{
  _iou:    "uuid-here",       # unique handle
  _driver: "file_store",      # which plugin
  _args:   { ... },           # driver-specific args
  _size:   8192               # bytes (optional, for indexing/quotas)
}

The _args hash is opaque to the index. Each driver defines its own arg schema:

Driver Args
file_store { path: "/abs/path" }
yaml_store { path: "/identities.yml", at: "alice.private_key" }
db_store { connect: "postgres://...", table: "blobs", id: "uuid" }
http_store { url: "https://...", method: "GET" }
ssh_store { host: "server", path: "/data/file" }
s3_store { bucket: "...", key: "...", region: "..." }

Driver Interface

A driver is a plugin that implements two methods:

module Wanderland
  module Storage
    module Drivers
      class FileStore
        # Write content, return an IOU marker.
        # The driver picks the args; caller doesn't dictate the path.
        def write(content, hint: nil)
          path = generate_path(hint)
          File.write(path, content)
          {
            "_iou" => SecureRandom.uuid,
            "_driver" => "file_store",
            "_args" => { "path" => path },
            "_size" => content.bytesize
          }
        end

        # Read content given the args from a marker.
        def read(args)
          File.read(args["path"])
        end
      end
    end
  end
end

Drivers register at process load time, same as boundaries:

Wanderland::Storage::Drivers.register("file_store", FileStore.new(root: "/data/files"))
Wanderland::Storage::Drivers.register("yaml_store", YamlStore.new)
Wanderland::Storage::Drivers.register("http_store", HttpStore.new)

Write Path

Storage::Registry.append(record)
  → SQLite driver receives the record
  → Check payload size against threshold
  → If small: store inline
  → If large: extract via configured driver
      → driver.write(content, hint: to_addr) → marker
      → replace payload leaf with marker
  → INSERT lean record into SQLite

Multiple IOU extractions per record are supported — IOU.extract walks the payload tree and replaces every large leaf with a marker. Each marker can use a different driver based on configuration (size, type, content pattern).

Read Path

Storage::Registry.at(addr)
  → SQLite returns lean record(s)
  → Caller receives records with IOU markers in payload
  → Optional: hydrate the record
      → Walk payload, find IOU markers
      → For each marker: lookup driver by name, call read(args)
      → Replace marker with content

Hydration is opt-in. Most queries don't need the full content — they just need to know what exists. Hydrate when you need the bytes.

URI Resolution

An IOU marker carries a URI, and a reader resolves that URI through an ordered chain of resolvers. Each resolver turns a URI into a next URI, a local file, or a stream. The loop runs until the value is concrete, and then hydration merges the bytes into the payload. A blob therefore lives in any store that a resolver can reach, and every service that holds the same resolver chain can read it.

This design keeps the existing marker fields and the existing drivers. A marker without a URI still reads, because the reader derives a URI from the marker's driver and arguments.

Marker forms

A file-store marker records an absolute path on the volume of the service that wrote it. This example comes from a Foxfire event record:

content:
  _iou: 28abbf2d-10db-417d-920c-29eb36720c91
  _driver: file_store
  _args:
    path: /data/blobs/28abbf2d-10db-417d-920c-29eb36720c91
  _size: 11160

A content-store marker records the SHA-256 hash of the bytes. The driver rebuilds the path from its own root directory:

content:
  _iou: 3f9a1c2b7e44d0a8...
  _driver: content_store
  _args:
    hash: 3f9a1c2b7e44d0a8...
  _size: 182044

The proposed marker carries one URI in place of the driver name and its arguments:

content:
  _iou: 3f9a1c2b7e44d0a8...
  _uri: cas:sha256:3f9a1c2b7e44d0a8...
  _size: 182044

The _iou handle and the _size field keep their meaning. A writer records _uri only. A reader accepts both forms.

The Resolution Chain

A resolver handles one URI scheme. It returns one of four results:

Result Meaning
a URI The value lives at another location. The loop resolves the new URI next.
a file The bytes are in a local file at this path. The value is concrete.
a stream The bytes are readable from this IO object. The value is concrete.
nothing The resolver does not handle this URI. The loop asks the next resolver.

The loop stops when a resolver returns a file or a stream. It stops with an error on a cycle or past a depth limit. When no resolver handles a URI, the marker stays in the payload unchanged, and the reader sees the IOU in place of the bytes.

direction: right
marker: "IOU marker\n_uri: cas:sha256:3f9a…"
cas: "cas: resolver"
content: ":content:sha256:3f9a…" { shape: cylinder }
s3: "s3: resolver"
file: "file: resolver"
hub: "hub: resolver"
stream: "Stream" { shape: oval }
local: "Local file" { shape: oval }
payload: "Hydrated payload"
marker -> cas: "resolves"
cas -> content: "returns the content address"
content -> s3: "returns its first reachable location"
hub -> s3: "returns s3://bucket/key"
s3 -> stream: "returns"
file -> local: "returns"
stream -> payload: "merges into"
local -> payload: "merges into"

The first set of resolvers:

The resolver order, the content-record prefix, and the fallback template come from the storage configuration. Two services with the same resolver configuration read the same markers.

Content Records

A content hash names the bytes, and a content record names where the bytes are. The record lives at :content:sha256:<hex>, with the type :type:content:

to: ":content:sha256:3f9a1c2b7e44d0a8..."
type: ":type:content"
payload:
  size: 182044
  locations:
    - s3://wanderland-content/sha256/3f/3f9a1c2b7e44d0a8...
    - file:///data/blobs/28abbf2d-10db-417d-920c-29eb36720c91

The writer orders the locations list. A reader takes the first location that a resolver can reach. A second copy of the bytes therefore gives a second source when the first store is not available.

A move of the bytes is a new version of the content record. The routine that moves a blob writes the bytes to the new store, then appends a version of the record with the new location. Every marker that names the hash follows the new location at once, and no marker changes. The record history keeps every earlier location, with its writer and its time, so the history of where the bytes lived comes from the ledger.

The content records sit under one prefix, :content:. The storage access levels apply to that prefix as to any other prefix.

A hydration that finds several markers in one payload reads their content records in one batch read, then streams each location. One batch read per payload replaces one ledger read per marker.

One Loop for Pointers and IOUs

A storage pointer (wanderland-core-storage-data-model) is a record of type :type:pointer whose points_at names another address. A ledger address is a URI in the ledger scheme. A pointer hop and an IOU hop are therefore the same step: resolve one reference to the next reference, until the result is concrete.

The resolution loop is one routine with one cycle check and one depth limit:

A pointer can therefore name a record whose payload holds an IOU, and an IOU can name a cas: location that names an s3: object. The reader follows the whole chain with one routine.

Markers Without a URI

The existing markers stay readable with no data conversion. Each existing driver supplies the URI for its own arguments:

Driver Arguments Derived URI
file_store {path: /data/blobs/<id>} file:///data/blobs/<id>
content_store {hash: <hex>} cas:sha256:<hex>

A reader that finds _driver and no _uri derives the URI, then resolves it through the chain. A derived file: URI resolves only on a host that mounts that volume.

The copy routine, POST /content/migrate (the content_migrate boundary), makes those blobs readable by every service. It changes no record that holds a marker, so a sealed chain stays intact. It runs inside the service that mounts the volume, and it takes a prefix:

A reader tries the marker's own location first. When that location does not resolve, the reader follows the alias :content:iou:<_iou> to the content record and its locations.

Lazy Hydration

A reader resolves a marker only when it reads that value. A listing, a graph walk, or a record read that does not read the large value never resolves the marker, and never fetches the bytes.

The read operations follow these rules:

Each resolver call happens at the moment of the read, as the configuration resolvers (!Pluck, !Sub) do. No step reads ahead, and no step caches bytes in the record. A record that holds a 256 KB log costs one ledger row in a list. It costs one fetch only when a reader opens the log.

Write Path and Threshold

A mount names one content sink. The sink stores the bytes and returns the marker's URI. A content-addressed sink writes each distinct value once, keyed by its SHA-256 hash, and returns a cas: URI. The hash also lets a reader verify the bytes after a round trip.

The extraction threshold is a mount setting. PostgreSQL stores a JSONB value of up to about 255 MB. It moves a value above about 2 KB out of the row into compressed side storage. A read of the record decompresses the whole value. The cost that matters is a read of a whole prefix, which loads every value under it. A threshold of 256 KB keeps logs, manifests, and documents in the row, and sends only large values to the content store.

storage:
  resolvers:
    - scheme: cas
      records: ":content:"                  # the content records, read first
      fallback: s3://wanderland-content/sha256/{prefix2}/{hash}
    - scheme: s3
    - scheme: file
    - scheme: hub
  mounts:
    ":":
      iou_threshold: 262144
      content_sink: cas

Existing Drivers Become Plugins

The current FileDriver and YamlStore get refactored into IOU driver plugins:

The SQLiteDriver becomes the universal index. Other backends become content stores accessed via IOU markers.

Configuration via Mount Spec

The storage mount spec declares which drivers extract for which prefixes:

storage:
  mounts:
    ":pki:":
      driver: sqlite
      path: ":memory:"
      content_drivers:
        - condition: { size: { gt: 4096 } }
          driver: file_store
          args: { root: "/var/wanderland/keys" }

    ":streams:":
      driver: sqlite
      path: "streams.db"
      content_drivers:
        - condition: { size: { gt: 1024 } }
          driver: file_store
          args: { root: "./content/streams" }

The SQLite driver consults the content_drivers list when appending. First matching condition wins. No match = inline.

Provenance Survives Driver Migration

Because the crossing record (with sig and trace) always lives in SQLite, you can:

The content is the leaf. The crossing is the root. The marker is the bridge.

Signed I/O

Every driver write is part of a crossing, which is signed by the writing identity. The marker preserves the link — you can follow the marker, read the content, and verify the original crossing's signature still matches the (canonical) record. Tampering with content in the backend doesn't break the index; it breaks the round-trip verification.

For stricter integrity: the marker can include a content hash (_sha256) so a rehydrate-and-verify round trip catches backend tampering immediately. The index sig covers the marker (which includes the hash). The hash covers the content. Two-layer integrity.

Implementation Path

Site Audit

wanderland.dev

oculus-view: fence: Fence execute bit is disabled. Use force=true to override.