diataxis: Explanation + Reference
style: ASD-STE100
status: design — the target shape of the WISP template store, not built
case: task-41245052-41a3-4b52-b793-e8c4fa52967c
tickets: ADV-1265, ADV-1460
feature: One template store in WISP — what a person authors, the records the store keeps, and how a run template compiles from them.
method: Derived from the WISP template sources and code (template_library.rb, template_expand.rb, offering.rb, template_store.rb, register.rb), an inventory of the 60 template entries, and the sim-planner prototype in vivarta-examples.
type_addr, a from_addr, and a payload.[Y]), a when:-guarded step]].A template is a workflow plus its metadata. A person authors one source YAML. The store keeps that YAML as small typed records at addresses. Shared blocks, such as the storage environment, are one record each, and a template refers to them. Every view of a template compiles from those records when a reader asks for it. The views are the run template, the offering, the contract, the parameter schema, and the tool schema.
Three writers put records on the same addresses. The importer writes what the YAML says, and the build writes what the pipeline source declares. A curator adds annotations. The overlay order sets which value wins. The from_addr of each record names who wrote it.
The diagram follows one template from authoring to the queue. Green marks a part that runs in production. Amber marks the new store work. Blue marks a new record family.
direction: right
classes: {
prod: { style.fill: "#e4efe0" }
dev: { style.fill: "#f4ecd4" }
future: { style.fill: "#e2e8f0" }
doc: { shape: document; style.fill: "#edeef1" }
}
yaml: Source YAML { class: doc }
build: Build schema { class: doc }
curator: Curator annotation { class: doc }
importer: Importer { class: dev }
records: Template records { shape: cylinder; class: future }
platform: Platform records { shape: cylinder; class: future }
compile: Compile on read { class: dev }
views: "Offering, contract, tool schema" { class: future }
y: "Run template [Y]" { class: future }
submit: Submit, hydrate, classify { class: prod }
queue: Queue { class: prod }
importer -> yaml: reads
importer -> records: writes
build -> records: writes
curator -> records: writes
records -> compile: feeds
platform -> compile: feeds
compile -> views: renders
compile -> y: renders
submit -> y: reads
submit -> queue: submits
One shape serves every template. The pipeline.source field is the only part that changes by kind.
| Key | Content | Who sets it |
|---|---|---|
id |
The job type. A submit names it. | The author. |
pipeline.source |
nf-core, repo, or image. |
The author. |
pipeline.ref, revision, commit |
The pipeline by reference, for nf-core and repo. Register pins the commit. |
The author, then register. |
pipeline.image, workflow |
The baked driver image and the workflow path, for image. |
The author or the build. |
size |
A size class from config/sizes.yml, such as md. |
The author. |
resources |
Per-axis overrides on the size class. | The author, optional. |
ports.inputs, ports.outputs |
Each file port with its container and its carrier. | The build declares the port. A curator adds the carrier. |
parameters |
Each scalar parameter: type, default, enum, range, description. | The build, then the author. |
offer_to |
The clients that see the template. | The author. |
ai_acceptance |
The AI data-sharing declaration. | The author. |
run |
Privileged run keys, such as secret_env or security_context. |
A platform author only. |
A template omits every value that the platform owns. Compile adds the storage environment, the mounts, the identity token, the entrypoint, and the run options.
id: demo
pipeline:
source: nf-core
ref: nf-core/demo
revision: 1.0.1
commit: 04060b4644d68649180db796e89a3db2a466a8d8 # register pins the tag
size: sm
ports:
inputs:
input:
container: file
staged_as: samplesheet.csv
required: true
carrier:
type: rnaseq_samplesheet
format: csv
description: The samplesheet (sample,fastq_1,fastq_2).
reads:
container: file-set
staged_as: reads/
required: true
carrier:
type: fastq
format: fastq.gz
description: The folder of FASTQ files the samplesheet references.
outputs:
multiqc_report:
container: file
pattern: multiqc/multiqc_report.html
carrier:
type: qc_report
format: html
offer_to: [datahub]
id: de_analysis
pipeline:
source: image
image: advaita/nf-ipg-core-de-analysis
workflow: /workflows/de_analysis/main.nf
size: md
ports:
inputs:
genes_counts:
container: file
staged_as: genes_counts.csv
required: true
carrier:
type: counts_matrix
id_ns: any # de_analysis does not read the gene ids
format: [csv, tsv, xlsx]
description: Raw counts matrix, genes on rows, samples on columns.
groups:
container: file
staged_as: groups.csv
required: true
carrier:
type: sample_groups
format: [csv, tsv]
contrasts:
container: file
staged_as: contrasts.csv
required: true
carrier:
type: contrasts_table
dialect: test_baseline
format: [csv, tsv]
outputs:
de_genes:
container: file
pattern: "{contrast}/de_genes.csv"
fanout:
key: contrast
from: contrasts
carrier:
type: gene_list_de
format: csv
keep:
id_ns: genes_counts # the output keeps the input's namespace
all_genes:
container: file
pattern: "{contrast}/all_genes.csv"
fanout:
key: contrast
from: contrasts
carrier:
type: gene_universe
format: csv
keep:
id_ns: genes_counts
parameters:
contrasts_test_col:
type: string
required: true
binds: contrasts # a column name in the contrasts port
contrasts_baseline_col:
type: string
required: true
binds: contrasts
sig_threshold:
type: number
default: 0.05
logfc_threshold:
type: number
default: 0.8
offer_to: [datahub]
Each record type is a type record with a schema. A type record lives under :types:.
| Type | Address pattern | One record holds |
|---|---|---|
:types:wisp-template |
:templates:<org>:<team>:<owner>:<id> |
The template head: id, size, offer_to, ai_acceptance, and the platform refs. |
:types:wisp-pipeline-source |
<template>:pipeline |
The source kind and its ref, revision, commit, image, and workflow. |
:types:wisp-port |
<template>:ports:<direction>:<name> |
One file port: container, staging, pattern, fan-out, and its carrier. |
:types:wisp-parameter |
<template>:parameters:<name> |
One scalar parameter: type, default, enum, range, binding, description. |
:types:wisp-plugin-pins |
<template>:plugins |
The Nextflow plugins the pipeline declares, with versions. |
:types:data-type |
:catalog:types:<name> |
One data type, its key facets, and its EDAM data id. |
:types:data-format |
:catalog:formats:<name> |
One file format and its EDAM format id. |
:types:wisp-env-block |
:platform:env:<name>:<env> |
One environment block for one environment. |
:types:wisp-mounts |
:platform:mounts:<name> |
One set of volume mounts. |
:types:wisp-identity |
:platform:identity:<audience> |
One projected identity token. |
:types:wisp-run-options |
:platform:run-options:<name> |
The Nextflow run options, such as the profile and the run name. |
:types:wisp-size |
:platform:sizes:<family>:<tier> |
One size class: cpu, memory, gpu. |
A template address is a containment path. The three scope segments are the owner's org, team, and user, and a platform template uses all in each. So a prefix read of :templates:advaita-internal: returns every template one org owns, and :templates:all:all:all: returns the platform set. A job_type is the last segment, and a pointer record at :templates:by-job:<job_type> resolves a submit to its template without a scan.
The decomposition is a stage of the template type's write path. The base type's write flow is parse, validate, diff, store. The template type's parser for its authored MIME is a boundary. It turns one authored document into the record set above. The validate stage then checks each record against its own type. The compile is the matching stage of the read path: each view in Stage 3 is a peek.view entry on the template type.
- addr: ":templates:all:all:all:de_analysis"
type_addr: ":types:wisp-template"
from_addr: "importer:config/dev/templates.datahub.yml"
payload:
id: de_analysis
origin: platform
size: ":platform:sizes:cpu:md"
offer_to: [datahub]
platform:
env: ":platform:env:storage:{env}"
mounts: ":platform:mounts:nextflow-work"
identity: ":platform:identity:datahub"
run_options: ":platform:run-options:kubernetes"
- addr: ":templates:all:all:all:de_analysis:pipeline"
type_addr: ":types:wisp-pipeline-source"
from_addr: "build:advaita/nf-ipg-core-de-analysis@sha256:5c1e…"
payload:
source: image
image: "789905347053.dkr.ecr.us-west-2.amazonaws.com/advaita/nf-ipg-core-de-analysis@sha256:5c1e…"
workflow: /workflows/de_analysis/main.nf
nextflow: 26.04.4
task_processes: [GENE_DE_ANALYSIS]
- addr: ":templates:all:all:all:de_analysis:ports:in:genes_counts"
type_addr: ":types:wisp-port"
from_addr: "build:advaita/nf-ipg-core-de-analysis@sha256:5c1e…"
payload:
direction: in
container: file
required: true
process_input: GENE_DE_ANALYSIS.genes_counts
- addr: ":templates:all:all:all:de_analysis:ports:in:genes_counts"
type_addr: ":types:poke"
from_addr: "importer:config/dev/templates.datahub.yml"
payload:
"$set":
staged_as: genes_counts.csv
description: Raw counts matrix, genes on rows, samples on columns.
- addr: ":templates:all:all:all:de_analysis:ports:in:genes_counts"
type_addr: ":types:poke"
from_addr: "curator:user:<sub>"
payload:
"$set":
carrier:
type: ":catalog:types:counts_matrix"
id_ns: any
format: [":catalog:formats:csv", ":catalog:formats:tsv", ":catalog:formats:xlsx"]
- addr: ":templates:all:all:all:de_analysis:parameters:sig_threshold"
type_addr: ":types:wisp-parameter"
from_addr: "build:advaita/nf-ipg-core-de-analysis@sha256:5c1e…"
payload:
type: number
default: 0.05
- addr: ":platform:env:storage:dev"
type_addr: ":types:wisp-env-block"
from_addr: "importer:config/dev/templates.datahub.yml"
payload:
DATAHUB_URL: http://advaita-drive.drive-qa.svc.cluster.local/api/pipeline
DATAHUB_RUN_ID: "<%= run_id %>"
DATAHUB_BUCKET: advaita-drive-qa
DATAHUB_OUTPUT_ROOT: out
DATAHUB_TOKEN_FILE: /var/run/secrets/app-identity/token
AWS_S3_ENDPOINT: https://cwobject.com
AWS_S3_PATH_STYLE: "false"
AWS_ACCESS_KEY_ID:
secret: datahub-qa-s3
key: s3-access-key
AWS_SECRET_ACCESS_KEY:
secret: datahub-qa-s3
key: s3-secret-key
AWS_REGION: us-east-1
- addr: ":platform:sizes:cpu:md"
type_addr: ":types:wisp-size"
from_addr: "importer:config/sizes.yml"
payload:
cpu: 2
memory_mb: 4096
| Writer | from_addr |
Writes |
|---|---|---|
| The importer | importer:<file> |
The values that the source YAML states. An unchanged file writes no record. |
| The build | build:<image>@<digest> |
The pipeline source, the ports, the parameters, the task processes, and the plugin pins, from the pipeline source. |
| A curator | curator:user:<sub> |
The carriers and other annotations, through the console. |
| Register | app:<app_id> |
A published template from POST /register. |
A record is a seed, and each later write is a per-field poke. The composed record is the seed plus its pokes in order. A later writer wins on the same field. The build alone creates a port. A curator annotation on a port that the build does not declare stays a gap finding, and compile shows no port for it.
The 33 copies of the storage environment block in the template files become one record per environment. The 58 identical nextflow-work mounts become one record. A change to the DataHub URL is one poke on :platform:env:storage:<env>.
Compile reads the template records, follows each platform ref, and makes one view per request. A cache holds each view until the store changes.
| View | Reader | Made from |
|---|---|---|
Run template [Y] |
The template library and classify. | The head, the pipeline source, the ports, the parameters, and the platform records. |
| Offering | GET /offerings and its clients. |
The head, the parameters, and the ports with their carriers. |
| Contract | The console and the planner. | Every port and parameter, each leaf with its source. |
| Parameter schema | validate_params and the Nextflow params file. |
The parameters and the input ports. |
| Tool schema | The LLM tool surface. | The offering, as a tool definition. |
The base config driver/nextflow.base.config declares the platform plugins in a plugins block. A plugins block replaces an earlier block, so a stock pipeline loses its own pins. Compile emits the run config with the union of the platform plugins and the pipeline's own pins. Each pipeline then runs the plugin versions it declares. The interim fix pins nf-schema@2 in the base config (foxfire PR #6).
# :templates:all:all:all:demo:plugins — from the build, read from the pipeline's nextflow.config
pins:
nf-schema: 2.5.1
# compiled run config for demo
plugins:
- nf-foxfire@0 # platform
- nf-advaita-datahub@0 # platform
- nf-schema@2 # the pipeline's own pin
id: de_analysis
when:
job_type: de_analysis
offer_to: [datahub]
driver:
image: 789905347053.dkr.ecr.us-west-2.amazonaws.com/advaita/nf-ipg-core-de-analysis@sha256:5c1e… # pipeline
command: [nf-driver-entrypoint] # platform run options
config_path: /workflows/de_analysis/nextflow.config # pipeline
workflow_path: /workflows/de_analysis/main.nf # pipeline
run_opts: [-profile, kubernetes, -name, "<%= run_id %>"] # platform run options
mounts: # platform mounts
- pvc: nextflow-work
path: /work
identity_token: # platform identity
audience: datahub
env: # platform env, this environment
DATAHUB_URL: http://advaita-drive.drive-qa.svc.cluster.local/api/pipeline
DATAHUB_RUN_ID: "<%= run_id %>"
DATAHUB_BUCKET: advaita-drive-qa
# … the rest of :platform:env:storage:dev
task_processes: [GENE_DE_ANALYSIS] # build
resources: # size class md
default:
cpu:
default: 2
min: 1
max: 4
memory:
default: 4096
min: 2048
max: 8192
gpu:
default: 0
params_style: flat
storage: automatic
parameters: # compiled parameter schema
$schema: http://json-schema.org/draft-07/schema#
title: Gene-only differential expression (DESeq2)
required: [genes_counts, groups, contrasts, contrasts_test_col, contrasts_baseline_col]
properties:
genes_counts:
type: string
format: file-path
path: genes_counts.csv
layer: process
sig_threshold:
type: number
default: 0.05
# … one property per port and parameter
The compiled [Y] has the same shape that classify reads in advaita-nextflow-router-data-model. Submit, hydrate, classify, and the queue stay as that node describes them, Stage A to Stage G.
| Step | Action |
|---|---|
| 1 | The importer reads config/templates.yml, config/<env>/templates.*.yml, and config/sizes.yml, and writes records. It sets origin: platform on each template it writes. |
| 2 | The importer factors the repeated blocks into platform records, and the template records refer to them. |
| 3 | A parity test compares the compiled [Y] of each job type with the assembled template of the old path. |
| 4 | The template library reads the store only. POST /templates/reload runs the importer again. |
| 5 | The ConfigMap path stays as an importer input until a later change removes it. |
The importer is idempotent. It diffs the composed record against the file and writes only the fields that changed. The old guard against a curated id that matches a platform id becomes a check on the origin field.