A Wanderland boundaries engine in vivarta-examples that plays the DataHub side of the storage contract (advaita-datahub-storage-data-model) with deterministic simulated data. It mints runs and fulfills every stage the compute plane calls — input:allocate (S3), files:allocate + :commit (S4/S5), the telemetry inbox (S6) — so the nf hub:// path factory and the driver are buildable end to end on a workstation, against Floci for the S3 bytes, with no wisp, no cluster, and no real DataHub. The stage it does not perform is submitting to wisp: it mints the run itself.
Source: vivarta-examples/sim-datahub. Code-doc mesh target for the @oculus:sim-datahub* annotations in that tree.
The model is a run as a tape. Every transaction against a run appends to an ordered tape; the run's state — the pipeline_job fields and the drive_file rows DataHub would hold — is a pure fold over that tape. Recording a driver's real transactions and replaying the tape lands in the same state, because every minted value (object keys, SSE-C keys, file ids) is a pure function of its inputs and already sits on the tape. That identity is the regression test: same transactions, same state. Rejected calls (the bounces) are recorded on the tape with their outcome, so a failure path replays as faithfully as a success path. The SSE-C keys it mints are real 32-byte AES-256 keys, so a driver's PUT/GET against Floci with them round-trips.
lib/sim_datahub/minter.rb. The deterministic source of every value DataHub mints: run ids (wisp's <lane>-<stamp>-<hex> shape), file ids, object keys in the <orgId>/<teamId>/<ownerId>/<fileId>/<hashed> drive layout, hashed names, per-file SSE-C keys (32 bytes → AES-256, base64), their base64 MD5, and the org-master-wrapped key stand-in recorded on the row. An input key is salted by the input_ref alone (a stored file has one key across every run that reads it); an output key is salted by its own object_key (unique per file, per run).
lib/sim_datahub/fold.rb. The pure reducer from a run's tape to its state. It reads no clock and mints no value — every value it needs is already on the tape. Only entries with outcome "accepted" change state; rejected entries fold to nothing. The inbox transitions mirror drive-service's FoxfireInboxService.applyEvent (run:started → RUNNING, task:* counters, run:completed → SUCCEEDED/FAILED on exit_status), with the terminal guard: a finished run drops every later event. The allocate/commit transitions build the drive_file rows — ALLOCATED at files:allocate, READY with size + metadata.sha256 at :commit.
lib/sim_datahub/run_store.rb. The process-local, mutex-guarded ledger of tapes, one per run_id — the only mutable state in the service. Every boundary appends a transaction here and reads the folded projection back. An idempotency index (key → run_id) gives duplicate-submission dedupe. load_tape seeds a whole tape under a run_id, the landing point for a replayed tape from the other side. A persistent DataHub keeps this in a database; the tape-and-fold shape is identical.
lib/sim_datahub/boundaries/create_run.rb. Mints a run and records run.created — the stage that stands in for wisp /submit + the poller. A repeat with the same idempotency_key resolves to the original run_id; creating a run whose run_id already exists is an idempotent no-op (never a second run.created, which the fold would read as a fresh run and wipe the tape's files). Validates the tenancy fence (org_id + owner_id required) → 422.
lib/sim_datahub/boundaries/input_allocate.rb. S3, the allocate-read. Resolves an opaque hub://<file_id> input ref (drive-service's contract — PipelineSubmitService emits hub://${file.id}, pinned by PipelineSubmitWiringIT) to the source object_key plus the file's SSE-C key, this call only — resolution returns an address and a key, so a hub:// ref is never pre-translated to a bare s3:// URI. The ref carries no org, so the tenancy fence is a lookup, not a parse: the file's org (from the FileRegistry when seeded, else the run's own tenancy for an unseeded ref) is checked against the run's tenancy org. A mismatch is a cross-tenant read → 403, recorded as a rejected transaction. A non-hub scheme (or a path-shaped ref) → 422. The SSE-C key is salted by the file_id, so a stored file resolves to the same key in every run that reads it.
lib/sim_datahub/boundaries/file_allocate.rb. S4 step 1. For each output, mints the file_id, an opaque object_key in the drive layout, and a fresh per-file SSE-C key. The friendly name is recorded but never appears in the object key. The row is ALLOCATED until :commit.
lib/sim_datahub/boundaries/file_commit.rb. S4 step 3 / S5. Reports {size, sha256}; the row becomes READY and the sha256 lands in metadata (no new column). Run-bound: a file this run never allocated → 404. Integrity cross-check: a supplied witnessed_sha256 that disagrees with the asserted sha256 → 409 (the driver's books and the store's witness must agree at the moment of record). Re-delivery of a READY file is a no-op.
lib/sim_datahub/boundaries/inbox_accept.rb. S6, the telemetry inbox. Files each nf-foxfire event on the run's tape; the fold applies the state machine. Best-effort like the real inbox: unknown run_id is a 200 no-op, a post-terminal event folds to no change. Accepts both projections — a flattened record (type/run_id top-level) or a raw signed envelope (fields inside a JSON payload string).
lib/sim_datahub/boundaries/run_state.rb. The folded projection of a run: the pipeline_job fields, the counters, and the drive_file rows, as DataHub would return them. The regression surface — the golden state a replay reproduces.
lib/sim_datahub/boundaries/run_tape.rb. The raw transaction tape for a run, every entry in arrival order including the rejected bounces. The input to replay and the payload sent to the other side.
lib/sim_datahub/boundaries/replay.rb. The regression engine. Folds a tape (a run's own, or one handed in inline) and returns the state; because the fold is pure and every minted value is on the tape, the folded state equals the state the tape was recorded from. With as_run_id it loads the tape under that id to read back; with target_url it POSTs the tape to another receiver — sim-datahub making a call to you, so the same transactions can be driven into a real DataHub or a second simulator.
spec/ (rspec over the booted engine via Rack::MockRequest) covers the stateful loop, the bounces, the record/replay identity, and the curated resolve→materialise path (picks model, StoreMap real-key resolution, the KEY_UNAVAILABLE fail-closed rung) — 37 examples. scenarios/scenarios.yml (bundle exec wanderland --type test config.yml) covers the per-route contract surface as an ordered build-up over the process-global tape — 25 scenarios (3 currently fail on pre-existing scenario-runner state leakage — run dedup + minted-file_id counter drift — not a boundary regression).
lib/sim_datahub/file_registry.rb. The process-local table of pre-existing input files — the drive_file rows a run reads from, keyed by file_id. A run's data map carries opaque hub://<file_id> refs, so input:allocate has no org to parse from the string and must look the file up. A file the registry does not know is treated as belonging to the run's own tenancy (faithful to buildSubmit, which resolves every input owner-scoped, so a run's inputs share its tenancy). Seed a file under a different org to exercise the cross-tenant fence.
lib/sim_datahub/boundaries/seed_file.rb. POST /files/seed — register a pre-existing input file (file_id → tenancy + name), the sim's stand-in for a file already in DataHub (what the user selected before submitting). Returns the hub://<file_id> ref to hand to a run's data map.
lib/sim_datahub/boundaries/import_dataset.rb. POST /import, intended as a CLI subcommand (a route IS a subcommand) — the standardized, manifest-driven way to stage a demo dataset. Replaces the ad-hoc pair demo/seed-bytes.rb (bytes) + demo/run-demo.py's registry-seed step. One manifest = one dataset: front-matter globals (dataset, tenancy, storage endpoint/bucket, seed_url) plus a files list, each with a byte source on disk resolved relative to the manifest.
Per file it does two toggleable jobs (--stage, --seed; both default true): (1) stage — PUT the bytes into object storage (Floci) at the MINTED drive key with the file's deterministic SSE-C key, using a bare Aws::S3::Client (lazily required; force_path_style + require_https_for_sse_cpk: false for Floci-over-HTTP, which the :aws adapter cannot set and which ships only in the deploy-only aws-pack); (2) seed — register a FileRegistry row via POST <seed_url>/files/seed through the core :http adapter, so a run resolves hub://<file_id>.
The invariant: both legs derive the object_key + SSE-C key from Minter directly (object_key(tenancy, file_id, stable_hashed_name), sse_c_key_b64(file_id)), so bytes land at EXACTLY the key + key-material input:allocate later returns for that registry file — the driver's GET round-trips by construction (pinned by spec/import_dataset_spec.rb, the importer ⇄ input:allocate agreement test). Validates every file before the first side effect (source present, tenancy fenced) so a bad entry imports nothing. New datasets are new manifests, no code change.
Run: bundle exec wanderland --type cli config.yml import --manifest demo/nf-demo.yml.
lib/sim_datahub/store_map.rb + data/store_map.yml. The static map from a DataHub file_id to a REAL object in CoreWeave/Floci object storage — a file_id's true object_key, its tenancy, size_bytes, and encryption posture (none | sse-c + sse_c_key_b64). This is what lets a test run read real test data (the demultiplex flowcell + samplesheet, the johnny smoke file) instead of the fictional keys the Minter produces.
A first-class file source, consulted in the same precedence as the FileRegistry (FileRegistry.find || StoreMap.input) by BOTH input:resolve (the fence + the volume sizing) and input:materialise (the real object location) — so a curated run can pick a shared real dataset with NO duplicate FileRegistry seed. This mirrors the real drive-service, where one DB row carries tenancy + size + location. A shared: true fixture is a public test object any run's tenancy may read (skips the cross-tenant fence), which is what lets a UI run under org "echo" read the johnny fixture. output_for(org) maps an org to its isolated per-run output offering (outputs → sim-runs/<run_id>/…). When a lookup misses, the boundary falls back to the minted-key drive layout — an unmapped run behaves exactly as before this file existed.
lib/sim_datahub/boundaries/resolve.rb. POST /api/pipeline/runs/{run}/resolve — the KEYLESS plan. Expands a curated run's picks (a file input's value is [{file_id, rel}] Drive references; a scheme scalar like hub:// passes through) into a materialise plan: target paths + file_ids + sizes + total_bytes, NO keys. Two passes — parse/reject the wire shape (UNKNOWN_FORMAT / MALFORMED_FILE_ID / UNRESOLVABLE_VALUE), then fence + size + target-assemble each pick (MISSING_REQUIRED_INPUT / WRONG_ARITY / FILE_NOT_FOUND / NOT_OWNED / NO_SIZE / TARGET_COLLISION). Each pick resolves by FileRegistry.find || StoreMap.input; a shared fixture skips the cross-tenant fence. The accepted plan is the stored per-run input ACL that input:materialise reads back. Keys are minted later, in one batch, at input:materialise — the plan is the security spine and never carries a key.
lib/sim_datahub/boundaries/input_materialise.rb. POST /api/pipeline/runs/{run}/inputs/materialise — the batch sibling of input:allocate: the one call the run-bound driver makes to key a curated run's WHOLE input tree before nextflow starts. No request body — the run's OWN stored plan (the last accepted resolve) is the authority, so there is no file list to tamper with. Every planned file is RE-FENCED against the run's tenancy (org AND owner, shared fixtures excepted); a plan is a snapshot and a file can be deleted or re-owned between resolve and exec. Precedence FileRegistry || StoreMap (no run-owned fallback — absent from both is FILE_NOT_FOUND, the snapshot cost, faithful to the real findByOrgIdAndId). A store-mapped file returns its REAL object_key verbatim + declared encryption (the driver GETs the actual Floci object); an unmapped-but-seeded file mints the drive-layout key (the sim's stand-in for storageKeyForBlob). All-or-nothing: any offender fails the WHOLE call and names every one, so the driver dies before nextflow, never a partial tree. A mapped sse-c object with no recoverable key → KEY_UNAVAILABLE (403), fail-closed. Keys are minted per call, never persisted. Mirrors drive-service PipelineInputMaterialiseService.