Background Workers

Settings → Background Workers (/admin/background-workers) is the control panel for the long-running processes the app depends on but does not run inside the web server. Each one is an ordinary Node script spawned as a detached child process, with its output redirected to a log file and its lifetime tracked in a BackgroundWorkerProcess row.

Facility admins only, like the rest of /admin. Every start, stop, pause and resume is attributed to the admin account that pressed the button, and the page header says so.

What is registered

The worker list is a static manifest (src/lib/workers/registry.ts), not something you configure in the UI. Four workers are defined; two of them are development helpers and are hidden entirely when the app runs in production.

Worker	Card label	What it does	Pause	Production
`pipeline-monitor`	Pipeline monitor	Reconciles `PipelineRun` status from scheduler state and Nextflow trace files when weblog callbacks are missing, delayed, or lost across a restart	No	Yes
`stream-monitor`	MinKNOW stream monitor	Watches the configured MinKNOW output directory and ingests FASTQ files into active stream runs	Yes	Yes
`stream-simulator`	Stream simulator (dev only)	Drips small FASTQ files into a watched directory to imitate a live run	No	Hidden
`discover-simulator`	Discover & Associate simulator (dev only)	Drops one batch of paired-end FASTQ files with Illumina-style names, then exits	No	Hidden

Both simulators run the same scripts/stream-monitor.ts entry point with different flags. Their output directories, barcodes and intervals are baked into the registry — the card says so — and changing them means editing the registry and restarting the dev server.

Why the pipeline monitor matters

A running pipeline reports progress to SeqDesk in two independent ways, and only one of them is guaranteed.

The Nextflow weblog (fast, but not always reachable)

Nextflow POSTs task events to the app as they happen. When it works, run status and per-step progress update within seconds.

The monitor (slower, but always available)

Every 15 seconds by default, the monitor lists every PipelineRun still in pending, queued or running and works out the truth from what is on disk and in the scheduler.

On a cluster, the weblog is frequently impossible. Compute nodes are commonly network-isolated from the application host, so Nextflow’s callback never arrives and the shared filesystem is the only channel back. On such a deployment the monitor is not a nicety — it is the only thing that ever moves a run out of running.

For each open run the monitor:

Parses the Nextflow trace file under the run folder and upserts one PipelineRunStep per step, resolving retries per task so a later success clears its own earlier failure without masking a genuinely failed sibling.
Asks the scheduler what actually happened — squeue first, then sacct for a finished job; for local runs, the recorded PID plus the run’s exit marker.
Reconciles the two. A terminal scheduler state always overrides a wedged trace that still claims 99%, which is what stops finished jobs from hanging as running forever.
On completion, ingests the run’s outputs before writing the terminal status. If that ingestion fails the run is deliberately held in a non-terminal state so the next pass retries — a run marked completed is never revisited.

A SLURM run whose trace only covers part of the pipeline is finalised from the scheduler state and the exit marker, never from trace progress alone. The early fast wave of a large pipeline otherwise reads as “everything known is done” in the gap before the next step is submitted, and the run would be declared complete after two steps of thirteen.

The pipeline monitor autostarts with the application. The server’s instrumentation hook calls it on every boot, including after an in-app update, and stops it again on a clean shutdown so it cannot outlive the release directory it was launched from. Set SEQDESK_DISABLE_WORKER_AUTOSTART=1 to opt out — for example when a process supervisor already owns it.

Its polling interval is PIPELINE_MONITOR_INTERVAL_MS, default 15000. The script also accepts --once for a single reconciliation pass, which is the useful form in a cron job or a debugging session.

The stream monitor

The MinKNOW ingest daemon polls the watched directory on an interval taken from STREAM_MONITOR_INTERVAL_MS, falling back to the configured poll interval and then to 5 seconds. It is the one worker that supports pause.

Pause is a soft flag, not a signal: it is stored as workerPause inside SiteSettings.extraSettings and re-read on every tick. While paused, the daemon skips ingest but keeps its watchers attached and does not mark skipped files as seen — so resuming picks up everything that arrived in the meantime instead of losing it. The log records PAUSED — ingest skipped, watchers stay attached and the matching RESUMED line.

Because the flag lives in the database rather than in the process, a paused worker that is restarted comes back still paused. Configuration for what it watches is on MinKNOW Stream, reachable from the card’s Settings button.

Telling whether a worker is alive

The panel refreshes every 5 seconds. Do not trust the stored status on its own — trust the reconciliation, which is what the page shows you.

Badge	Meaning
Running	A tracked PID exists and the process answers a liveness probe
Paused	Running, with the pause flag set
Stopping…	`SIGTERM` sent, inside the grace period
Stopped	Not running, or a stale row that has just been reconciled
Error	The process exited with a non-zero code
Zombie	A row that outlived its process; cleared to Stopped on the next read

Every page load asks the server to reconcile each worker: it takes the newest row for that worker and probes the PID directly. A row claiming Running whose process is gone is corrected to Stopped on the spot, so the panel cannot show you a daemon that died in the night as healthy. Rows are never trusted alone, because a Next.js restart loses the in-process exit listener that would otherwise have marked the row.

Beyond the badge, each card gives you:

pid, how long ago it started, and which admin started it. “Never started” means no row exists at all.
The exact command line, including baked-in arguments.
The last error message, if the process died with a signal or a spawn failure.
Show log — a live tail of the worker’s log file, refreshed every 4 seconds while open. Logs are written to logs/<worker>-<pid>.log in the application directory; the panel reads up to 100 lines by default and the API caps a request at 2000.

Confirm from the panel

Open Settings → Background Workers and read the badge. It is already reconciled against the real process.

Confirm from the log

Press Show log. Both daemons announce their poll interval on startup — [pipeline-monitor] running every 15000ms and [stream-monitor] running every 5000ms — and every line they write afterwards carries the same prefix. The first two lines of any log are written by SeqDesk itself and record the PID, the hostname and the exact command.

Confirm from the host

If you have shell access, ps -p <pid> on the PID shown by the card settles it. The panel’s own liveness check is exactly this probe.

Starting and stopping

Only one instance of a worker may run at a time. Start refuses with “<name> is already running (pid=…). Stop it first.” when a live process is already tracked; a stale row for a dead process is cleared automatically and a fresh worker is spawned. Stop sends SIGTERM, waits ten seconds, then sends SIGKILL.

Workers are spawned detached, so they survive a Next.js reload — but a worker started here is a child of the application and inherits its environment, including DATABASE_URL.

On a production deployment behind PM2, systemd or another supervisor, leave these stopped and let the supervisor own them. Starting a worker here while a supervisor also runs one gives you two instances racing on the same database rows. The same applies to the autostarted pipeline monitor: if your supervisor runs it, set SEQDESK_DISABLE_WORKER_AUTOSTART=1.

What breaks if a worker is not running

Worker down	Symptom
`pipeline-monitor`, cluster deployment	Runs stay at running forever. Nothing finalises them, outputs are never ingested, and no artifacts or read writebacks appear
`pipeline-monitor`, local deployment	Usually invisible while weblog callbacks arrive — until a run’s callback is lost or the app restarts mid-run, at which point that run is stuck with no way to recover on its own
`stream-monitor`	New FASTQ files pile up in the watched directory and are never ingested; the Stream view stops advancing
Two `pipeline-monitor` instances	Both reconcile the same rows and both attempt output ingestion. Ingestion is idempotent, but the duplicated work is pointless and the logs become unreadable

Troubleshooting

Symptom	Cause	Fix
Start answers `is already running`	A live process is tracked	Stop it first, or leave it — one is what you want
Start fails with `ENOENT` or `no PID assigned`	The runner could not be resolved from the application directory	Check that the install is complete and start the app from its own directory
Card says Never started but pipelines finish anyway	Weblog callbacks are arriving; the safety net has simply never been needed	Start the monitor anyway before running on a cluster
Badge flips to Stopped by itself	Reconciliation found the PID gone	Read the log tail for the exit reason, then restart
Worker restarted but still ignores new files	The pause flag is in the database, not the process	Press Resume
Simulator cards are missing	They are `devOnly` and hidden in production	Expected; they exist for development only
Log panel says `no log file yet`	The worker has never started on this host	Start it once
Runs finish on the cluster but stay running in SeqDesk	The monitor is not running, or cannot reach `squeue`/`sacct`	Start it and confirm the scheduler commands work from the application host