Wanderland

advaita-nextflow-event-system

Every meaningful thing that happens in a Nextflow run — a task starting, a file written, a QC verdict, a verse sung — becomes a signed crossing on the run's append-only trace in foxfire. This node is the model of that event system: the four things that produce events, the one path that signs and ships them, the store that seals them into a tamper-evident chain, and the state machine a run/task can drive through them.

Sits under advaita-nextflow-router-data-model §Stage G, which names the three ledgers (plan · placement · trace) joined on run_id. This is the trace ledger, expanded — how it fills. The signing/shipping core is nf-foxfire; the receiver is advaita-service-foxfire.

The four producers

All four run on the driver (head) pod, where the signing key lives. A task pod never holds the key — it only prints (see the poller). Each producer hands a (type, payload) to the one shared Emitter, which signs and ships it.

Producer Where Emits Example types
Trace observer (FoxfireObserver, TraceObserverV2) nf-foxfire the Nextflow lifecycle, with each task's resource trace (the cost fields) :events:run:started, :events:task:submit/start/complete/cached, :events:run:completed
foxfireEmit (workflow function) nf-foxfire workflow-authored domain crossings, from the DSL (head-side orchestration points) :events:fastqc:started/report/completed
The poller (ProgressTailer) nf-foxfire task-authored crossings, live — markers a task prints to stdout :events:fastqc:progress/module/summary, :events:synth:lyric
Storage observer (DatahubObserver) nf-advaita-datahub storage crossings, on resolve/publish :events:input:resolved, :events:file:written

The split that matters: foxfireEmit is head-side (callable only in the workflow/DSL, where the plugin function and the key exist). The poller is the task-side path — the only way bytes emitted inside a task pod's script block become signed events, because the task has no key and no plugin runtime.

The poller — task-side live progress

A task prints marker lines to stdout:

::foxfire:: <type> <json>

ProgressTailer, on the head, tails each task's .command.out off the shared nextflow-work volume as it is written (polled), parses each marker, and re-emits it through the head's Emitter — signed, folded onto the run's trace, fanned out like every other event. The task only prints; the head signs. Progress streams while the task runs, not batched at the end.

Authoring a marker (from a process script/shell block):

# one line per event; the payload is a JSON object. An embedded newline stays the
# JSON \n escape (printf %s, not %b) — the tailer reads line by line.
printf '::foxfire:: :events:fastqc:module {"module":"%s","status":"%s"}\n' "$m" "$s"

Config (the foxfire scope, all optional):

Key Default Meaning
progressTail true tail task stdout for markers and re-emit them signed
progressMarker ::foxfire:: the marker prefix
progressLogFile .command.out the per-task file tailed
progressIntervalMs 500 poll interval

Worked examples: fastqc streams scan progress + the 11 module PASS/WARN/FAIL verdicts + a summary; hole-in-my-bucket sings each verse as :events:synth:lyric in time with the tempo (karaoke over the trace).

Sign → ship → fan out

One Emitter (per run, built by FoxfireRuntime) is shared by all four producers, so every crossing signs with one key and ships through one queue.

Store + seal chain

foxfire appends each inbound event verbatim as an immutable row at :events:event:<run_id>:<seq>, seq = the count at insert. Rows form a tamper-evident seal chain: each seals over its (seq − 1) predecessor's signature (seq 0 links to a per-service genesis). Two read surfaces:

The invariant that keeps it intact: record_event must serialize its seq = length; append. A burst of task-progress crossings interleaving with the storage observer's file:written events once let two events share a seq and skip the next — which snapped the chain (broken_at at the collision). It is now guarded by a process-level write mutex (foxfire is the store's single writer by design — replicas: 1). Regression: 60 concurrent record_events must fold to a gap-free 0..N-1.

The state machine a run/task drives

Events are recorded verbatim, but only the lifecycle ones drive state. Two independent folds consume the same stream:

                     :events:run:started ──▶ RUNNING
run lifecycle  ──▶   :events:task:submit/start/complete  (counters: submitted/running/done)
                     :events:run:completed ─▶ SUCCEEDED | FAILED  (on exit_status)

Choosing a producer (author's guide)

You want to signal… From Use
a workflow-orchestration milestone (before/after a stage) the DSL / workflow {} foxfireEmit(type:, payload:)
progress or a result from inside a task, live a process script/shell block a ::foxfire:: marker → the poller
a resolved input / a published file — automatic (the storage observer)
run/task lifecycle + cost — automatic (the trace observer)

Everything lands on the same signed, sealed trace; the only choice is which side of the head/task boundary you're on.