Companion to advaita-declarative-pipeline-publishing. This is the walkthrough we hand a developer (and, later, a scientist). If any step here feels heavy to us, it will feel heavy to them — that's the signal to cut, not to explain harder.
Goal: get nf-core/rnaseq into DataHub's Run Workflow catalog. No image build, no driver repo, no DataHub release.
Time: one file, one command, one refresh.
You need:
nf-core/rnaseq, 3.26.0.nextflow_schema.json (every nf-core pipeline ships one) and pick the handful a user should set. You do not redeclare them — you reference them; wisp checks your list against that schema at publish time.POST /register against wisp). Nothing else — no Docker, no Kotlin, no ECR.That's the whole prerequisite list. If it grew past this, the design failed.
One file, and a curator writes it — not the end scientist. (The scientist is the biologist who later picks files in the catalog; the curator is whoever turns a pipeline's ~133 params into the handful worth showing.) It's a few fields plus a size hint, and it looks like a wisp template because it is one:
# rnaseq.yml
id: rnaseq-star-salmon
pipeline:
nf_core: nf-core/rnaseq # or `repo: advaita-corporation/<fork>` for a fork
revision: "3.26.0" # you type a tag; the platform pins it to a commit SHA for you
size: large # a t-shirt hint — STAR's index is memory-hungry
parameters: # what the user sees in the catalog form
properties:
input: # a FILE the user picks (format: file-path)
{ type: string, format: file-path, pattern: "\\.csv$",
description: "Samplesheet — sample, fastq_1, fastq_2, strandedness" }
genome:
{ type: string, enum: [GRCh38, GRCm39], description: "Reference genome" }
aligner:
{ const: star_salmon } # you pin this; the user never sees it
required: [input, genome]
You wrote id, pipeline, size, and parameters. You did not write: an image, a service account, credentials, Kubernetes, tenancy ids, or per-process CPU/memory — the platform fills those. Your job is choosing which of the pipeline's params a scientist should see (here, 3 of rnaseq's 133) and which files they pick. That curation is the product.
Two levers do most of the work, both plain JSON Schema: const pins a value and hides the control (aligner: star_salmon), and narrowing an enum turns a free-text field (genome) into the few references we actually host.
The scientist-facing thing is a publish job with the wisp URL and credentials already wired — they run it, they don't type an endpoint. Under the hood it's one POST:
# what the generated publish job runs for you:
curl -s -X POST http://wisp.vivarta-drivers.svc.cluster.local:9321/register \
-H 'content-type: application/json' \
--data-binary @rnaseq
Publishing does two things and tells you if either fails:
parameters against nf-core/rnaseq's own schema — expose a param the pipeline doesn't declare, or give it a bad type, and you get a readable 422 here, at publish time, not a silent no-show in the catalog later.id updates it in place — edit rnaseq.yml, publish again, done.A clean publish returns the registered template and a link to it in the catalog.
DataHub reads registered templates on its refresh and folds them into Run Workflow. Confirm it's live:
curl -s http://wisp.vivarta-drivers.svc.cluster.local:9321/templates | jq '.value[] | select(.id=="rnaseq")'
If it's in that list, it's in the catalog. DataHub validates the offer on its side (reject-and-keep-previous, so a bad entry can never blank the catalog) and renders your parameters block as the input form.
The user picks RNA-seq in the catalog, sets genome: GRCh38, leaves aligner on its default, and hits run. DataHub submits:
curl -s -X POST http://wisp.vivarta-drivers.svc.cluster.local:9321/submit \
-H 'content-type: application/json' \
-d '{
"job_type": "rnaseq",
"params": { "genome": "GRCh38", "aligner": "star_salmon" },
"data": { "reads": "hub://<file_id>" },
"tenancy": { "org_id": "advaita", "owner_id": "u-31f2" }
}'
wisp matches your record, cold-starts the sizing from a safe default (and right-sizes from the run's own trace next time), and the shared base image runs nextflow run nf-core/rnaseq -r 3.26.0 stock — with params.genome and params.aligner at the top level where the pipeline actually reads them. No rnaseq-specific image ever existed.
rnaseq.yml, re-run Step 2. It's an upsert.rnaseq.yml in your repo and re-publish the old one).Walk it and mark each step. After the three-field cut, here's where it stands:
step 1 (write the doc): three fields you already know — id, pipeline, parameters.
no sizing, no processes, no image, no tenancy. "naming a pipeline." ✓
step 2 (publish): one job, one readable error on failure. the scientist never types
the endpoint — the generated job carries the URL + creds. ✓
step 3 (verify): the publish response hands back the catalog link. ✓
step 4 (run): the user's job, not the publisher's. ✓
Settled by the cut (decided in advaita-declarative-pipeline-publishing):
resources / task_processes. The class cold-starts from a default; processes are discovered from the first run's trace. This is what shrank Step 1 to three fields.org_id/team_id/owner_id/id — the same tuple used everywhere else — so it scopes all the way down to one user without a later migration.Still open before we hand this out:
id need a separate display title from the matching key? (parameters.title covers the form header; the catalog card may want its own.)make publish, a repo CI step?If Step 1 reads as "naming a pipeline," we cut enough. It does.