Monitoring & DAG Visualization

SeqDesk provides real-time monitoring of pipeline runs through Nextflow’s weblog integration and trace file parsing.

Real-Time Monitoring

Weblog Integration

The primary monitoring method uses Nextflow’s weblog feature. When a pipeline runs, Nextflow sends HTTP events to SeqDesk’s weblog ingest endpoint:

POST /api/pipelines/weblog?runId={runId}&token={secret}

Authentication is via the runId and token query parameters (a shared secret), not a session or request body. The weblog secret is mandatory:

• If no weblog secret is configured, the endpoint fails closed with 503 — an unauthenticated state-mutating webhook would otherwise let anyone drive run state.
• If the supplied token does not match the configured secret, the endpoint returns 403.

Each event updates:

• Run status — pending, queued, running, completed, failed
• Step status — which Nextflow processes are running/completed
• Progress — overall percentage (0–100)
• Output tail — last lines of stdout/stderr

SeqDesk keeps the most recent 100 events per run (older events are pruned as a ring buffer) and de-duplicates events that arrive within a 2-second window.

Workflow-Completion Finalization Hold

When Nextflow reports workflow_complete/workflow_finish but SLURM still reports the job as active (or run outputs have not yet materialized), SeqDesk holds the run as “Finalizing…” and caps progress at 99% rather than flipping straight to completed. The run is only marked completed once the scheduler is idle and the expected outputs have been resolved, which avoids showing a run as done before its files are actually written.

Trace File Fallback Monitor

If weblog events are unavailable or delayed, SeqDesk can monitor runs from the trace.txt file that Nextflow generates plus the SLURM queue state. A background monitor can be run:

# Continuous monitoring
npm run pipeline:monitor
 
# Single check (for cron jobs)
npm run pipeline:monitor:once

The trace monitor reads process execution data, reconciles it with the scheduler, and updates run status accordingly. It applies the same finalization hold described above, so a run is not marked completed until its outputs have materialized.

Pipeline Run Statuses

DAG Visualization

The pipeline definition file (definition.json) describes the workflow as a directed acyclic graph (DAG). SeqDesk renders this as an interactive visualization showing:

• Each Nextflow process as a node
• Dependencies between processes as edges
• Current status of each process (color-coded)
• Progress through the overall workflow

Process Steps

For the MAG pipeline, the main process steps are:

Event Feed

Pipeline events are logged in the PipelineRunEvent table and shown in the monitoring UI:

Weblog Setup

To enable real-time monitoring:

1. Go to Admin → Pipeline Runtime
2. Set the Weblog URL to your SeqDesk instance’s public URL + /api/pipelines/weblog
3. Set a Weblog Secret — this is required. Until a secret is set, the weblog endpoint rejects every event with 503, so runs will not report real-time progress over the weblog (they fall back to the trace monitor).
4. The URL and per-run token are automatically included in the generated launch command.

The weblog URL format:

https://your-seqdesk.example.com/api/pipelines/weblog