Monitoring a Run

SeqDesk tracks a running pipeline from three independent sources and reconciles them. Understanding which source produced the status you are looking at is the difference between “this run is stuck” and “this run is fine, the callback path is not”.

None of them is trusted unconditionally. A wedged trace can report “running 99%” long after the job finished; a weblog event can arrive after the run is already terminal; a scheduler can still show a wrapper job alive while the workflow itself is done. The reconciliation rules below are the result of fixing each of those in turn.

What you see while a run is in flight

Open a run from the Pipeline Runs table on either Analysis surface, or from the global Analysis page. The run detail page (/analysis/{id}) has three tabs:

While the run is active, the page auto-syncs every 15 seconds — it calls the run’s sync endpoint, then refetches. Two badges tell you how healthy that loop is:

• Live — the last update is under 60 seconds old.
• No updates — the last update is over 5 minutes old. The run may still be fine (a long single process produces no events), but this is the first thing to check when progress looks frozen.
• Sync warning — auto-sync itself failed. Hovering shows why; HTTP 403 means your role cannot drive sync, so the timestamps you see may be stale.

Pipeline Steps is a vertical timeline, one row per step defined in the package’s definition.json, with a status icon, the step name and its duration. It is not an interactive graph — the graph view of a pipeline’s DAG lives in Admin → Pipelines, in the package dialog’s Workflow Steps tab, and shows the definition rather than a specific run.

Logs tails logs/pipeline.out and logs/pipeline.err from disk (falling back to the cached tail stored on the run), showing the last 100 lines by default.

Run states

PipelineRun.status is one of six values.

completed, failed and cancelled are terminal. Both the weblog handler and the sync path guard on that: a late duplicate workflow_complete, or a trace task that still reads “running”, can never resurrect a terminal run or clear its completedAt.

Alongside the status, the run carries currentStep — a human label rather than an enum. The ones worth recognising:

statusSource records which of weblog, trace, queue, process, launcher or manual last set the status. It is shown in the debug bundle and is the fastest way to answer “who decided this run was done?”.

The weblog path

When a weblog URL is configured, the generated nextflow.config enables Nextflow’s weblog feature and points it at the app with the run id and a shared secret as query parameters:

POST /api/pipelines/weblog?runId={runId}&token={secret}

Authentication is those two query parameters — not a session, not a body field. The endpoint fails closed:

• No weblog secret configured → 503 Weblog secret is not configured. An unauthenticated state-mutating webhook would let anyone drive run state, so the endpoint refuses to work at all until a secret exists.
• Wrong token → 403 Invalid token.
• Unknown runId → 404 Run not found.

Each accepted event updates the run’s status, current step, progress and step rows, and is stored as a PipelineRunEvent. Two limits keep the table bounded:

• Events identical in type, process, step, status, message and payload within a 2-second window are treated as duplicates and not stored twice.
• Only the most recent 100 events per run are kept; older ones are pruned on each write.

Event timestamps are sanity-checked against the receive time: more than 6 hours in the future or 30 days in the past, and the receive time is used instead.

Progress is computed as completed steps over the package’s total defined steps, capped at 99% while running, and it never decreases — a late event cannot drag a run’s progress backwards.

The trace and scheduler fallbacks

Weblog events are not always possible. The most common reason is network isolation: on many clusters the compute nodes cannot reach the application server at all, only shared storage. Such a run has lastWeblogAt empty for its whole life, and every status update comes from the trace and the scheduler.

Trace. SeqDesk parses trace.txt, groups tasks into steps using each step’s processMatchers, and resolves them with two deliberately different rules:

• Retries of the same task resolve to the last word: a later COMPLETED or CACHED attempt clears that task’s earlier FAILED. Otherwise a successfully-retried step would be pinned to failed forever.
• Distinct sibling tasks do not: if any distinct task failed, the step is failed, even when a sibling succeeded. Otherwise a genuinely failed task would be masked and the run falsely reported complete.

A CACHED task — reused by -resume — counts as completed, not pending. In the background monitor’s accounting a SUBMITTED task counts as pending rather than running, because a job queued in the scheduler is not yet executing.

Scheduler. For SLURM, squeue is asked first (live state) and sacct second (accounting, for jobs that already left the queue). For local runs the process id recorded as queueJobId = local-<pid> is probed. A terminal scheduler state overrides a non-terminal trace status, which is what unsticks a run whose trace wedged at “running 99%”. A terminal trace status is never overridden.

The background monitor. A daemon can drive the same reconciliation without anyone having the page open. It only ever selects non-terminal runs:

npm run pipeline:monitor        # continuous, default every 15 s
npm run pipeline:monitor:once   # single pass, for cron

Set PIPELINE_MONITOR_INTERVAL_MS to change the interval.

How a run is finalized

This is the part with the most hard-won behaviour, because the failure mode — marking a run completed before its outputs exist — is silent and permanent (the monitor never revisits terminal runs).

Outputs are ingested before the terminal status is written. If ingestion throws, the run is held at running with Finalizing outputs... and 99% so the next pass retries. Re-ingestion is idempotent: artifacts are unique per (run, path) and assemblies and bins per (run, sample, file).

A still-active scheduler job outranks a completed workflow. If Nextflow reports workflow_complete while SLURM still reports the job active, the run stays running, the label becomes Finalizing…, progress is capped at 99%, and completedAt is left null. Only when the scheduler is idle and outputs have resolved does it flip to completed.

A late run-scoped output is waited for. Run-scoped outputs — a summary TSV, typically — are written by the last process, after the per-sample files. On shared NFS they may not be visible at the instant the run flips. Discovery re-scans up to three times, one second apart, while a declared run-scoped output is still missing.

Trace progress alone never proves completion. overallProgress is computed over the tasks already in the trace, so it reads a trivial 100% in the gap after the first fast process finishes and before the next is submitted. For a package that ships no step definitions, that gap can last minutes while Nextflow builds a conda environment. Such runs are finalized only from positive exit evidence: the canonical exit marker in logs/pipeline.out, or a terminal scheduler state.

For runs with step definitions, “every step in the trace is done” is not enough on SLURM. A SLURM run must also have completed at least as many steps as the package defines. Otherwise an inline-executor job that ingests reads in a fast first wave reads as “all known steps done” and gets finalized after 2 of 13 steps, while the job is still running.

The signal all of this leans on is the exit marker written by the run wrapper’s EXIT trap:

Pipeline completed with exit code: 0 at Wed Apr 15 11:42:03 CEST 2026

SeqDesk matches only that exact phrasing — deliberately, not generic exit code: N substrings, because Nextflow streams task error reports and conda solver output into the same file while the run is still executing, and any of those can contain such a substring.

Cancelling a run

Cancel on the run detail page, or Stop run in the Pipeline Runs table, is available while the run is pending, queued or running. Anything else returns Cannot cancel a completed or failed run (400).

What happens:

• Local runs are killed by signalling the process group (SIGTERM), falling back to the single PID.
• SLURM runs are cancelled with scancel.
• The run is written as cancelled with statusSource: manual.
• If the kill itself fails, the run is recorded as failed rather than cancelled, so the record does not claim a clean stop that did not happen.
• The write is guarded to non-terminal states. If the run completed between the status read and the write, the response reports the real status with alreadyFinalized: true and the UI says Run already finished. A genuine completed outcome — and its ingested outputs — is never clobbered.

Cancelling does not delete the run folder or any artifacts already ingested. See Deleting a run.

Troubleshooting

No progress at all; lastWeblogAt never set. Either the weblog is unconfigured, or the compute node cannot reach the app. Both are normal on isolated clusters — the run still progresses via the trace and the scheduler, just with coarser granularity. Confirm by checking that trace.txt is growing.

Every weblog event rejected with 503. No weblog secret is set. Fill in Weblog Secret under Weblog Setup; the endpoint refuses events until then.

Run sits at queued forever on SLURM. Look at the scheduler reason on the Status tab. (PartitionTimeLimit) means the requested time limit exceeds what the partition allows — remember the setting is in hours. A per-user submission QOS limit can also pin the only slot with a leftover job; cancel stale pending jobs before resubmitting.

Run shows “Finalizing…” and does not move. The workflow says it is done but the scheduler still reports the job active. This is expected for a short while while an inline wrapper winds down. If it persists, check the scheduler directly — the job may be stuck in COMPLETING.

Run shows “Finalizing outputs…” repeatedly. Ingestion is failing and being retried. Check the server log for post-completion resolution errors, and check that the declared output paths exist under the run folder’s output/.

Run marked failed with no failed task in the trace. The wrapper exited non-zero before any task ran — usually a Conda environment build failure, or a missing reference database. Read logs/pipeline.err; the failure summary shown on the Status tab is extracted from the Command error: block or the first error-looking line of the log tail.

Run detail page is 403 for the researcher who owns the study. Runs are invisible to non-admins until published. See Publishing a run.

You need everything at once. A facility admin can fetch a debug bundle for a run — run fields, paths, file stats, scheduler output and log tails — as JSON or, with ?format=text, as a pasteable plain-text report:

GET /api/pipelines/runs/{id}/debug?format=text

Weblog Setup

Real-time monitoring needs two settings under Admin → Pipeline Runtime, in the Advanced Configuration section:

1. Nextflow Weblog URL — your instance’s externally reachable base URL plus /api/pipelines/weblog, for example https://seqdesk.example.org/api/pipelines/weblog. Use Test next to the field to verify the endpoint answers with the secret you entered.
2. Weblog Secret — a shared token. This is required; without it the endpoint rejects every event with 503 and runs fall back to trace monitoring.

Press Save Runtime Settings. The URL, the run id and the token are then injected into each run’s generated nextflow.config automatically — there is nothing to configure per run.

Related pages

• Running a Pipeline — what produces the run folder and the exit marker.
• Results & Outputs — what ingestion actually writes.
• Administration → Pipeline Runtime — the settings referenced above.
• Adding Custom Pipelines — declaring processMatchers so trace progress can attribute work to your steps.