Stream Mode Troubleshooting

Stream mode is beta and under active development. Several of the symptoms below are known bugs and unfinished work rather than misconfiguration, and they are called out as such — they are being worked on, so a symptom listed here may simply be gone after an update. If something on this page does not match what you see, trust the daemon’s log over the page, and check the known limitations for the current state.

Start by deciding which part failed

Stream mode has three moving parts, and almost every problem is one of them being silent.

Is the daemon running? Settings → Background Workers, or the inline badge next to Start receiving. Stopped means nothing will ever be ingested, regardless of how correct everything else is.
Is it paused? A paused worker keeps its watchers and skips every file. The card shows a Paused badge and the daemon logs PAUSED — ingest skipped, watchers stay attached.
Is MinKNOW writing where you think? Check that <run folder>/fastq_pass/<barcode>/ actually contains .fastq or .fastq.gz files, from the daemon’s host, as the daemon’s user.

The start request was refused

Every message below comes from the start endpoint verbatim.

Message	Cause	Fix
`outputDir is required`	The field was empty	Enter the run folder
`MinKNOW outputRoot is not configured (Application Settings → MinKNOW Stream)`	No output root saved	Set it on the settings screen
`outputDir must be an absolute path`	A relative path was sent	Use the full path
`Configured outputRoot must be an absolute path`	The saved root is relative	Fix the setting
`outputDir does not exist or is unreadable: …`	The run folder does not exist yet, or the web application’s user cannot read it	Wait until MinKNOW creates it; check permissions
`Configured outputRoot does not exist or is unreadable: …`	The root is wrong, or its mount is not present on this host	Fix the setting, or mount the share
`outputDir resolves to <X> which is not under the configured outputRoot <Y>`	After following every symlink, the directory lands outside the root	Point at a directory genuinely under the root, or widen the root
`Another active stream (id=…, order=…) is already watching <dir>. Stop it first.`	One directory, one active stream — including from a different sequencing order	Stop the named run
`Another active stream is already being started for <dir>. Stop it first or try again.`	Two starts raced; the database serialised them and this one lost	Reload; if no stream exists, try again
`Only facility admins can manage sequencing data`	Wrong role	—
`Sequencing data management is disabled in the public demo.`	Demo session	Reading is allowed, starting is not
`Failed to start stream` (500)	Anything unexpected — including a sequencing order id that does not exist, which fails on the foreign key rather than as a 404	Check the server log

The rejection message names the resolved path, not what you typed. That is usually the fastest way to discover that a path you believed was inside the root is a symlink to somewhere else.

The stream is active but nothing arrives

Work down this list.

Check	Detail
The daemon is running and not paused	See above
`fastq_pass/` exists inside the run folder	The watcher is attached to that subfolder specifically, not the run folder
The files are basecalled FASTQ	Only `.fastq`, `.fastq.gz`, `.fq`, `.fq.gz` are considered — `pod5/` is ignored entirely
You are not waiting on `fastq_fail/`	Failed and skipped reads are never ingested
The daemon’s user can read the tree	The Test connection check runs as the web application’s user; a pass there does not prove the daemon can read the folder
The share emits filesystem events	On NFS and SMB, inotify and FSEvents often do not fire. Turn on `usePolling`, and raise `stabilityThresholdMs` for a slow share
The daemon log	`attached watcher for stream <id> → <path>` on attach; `outputDir missing` if the directory vanished

Files arrive but nothing links to a sample

Symptom	What happened
By barcode shows a row marked unmapped, with files but zero reads	The folder name is not a key in this stream’s barcode map. Unmapped files are logged and counted as files, but never parsed or linked
An `ERROR` event says `mapped sampleId not found in this order`	The mapping points at a sample that belongs to a different sequencing order
An `ERROR` event says `could not parse barcode from path`	The path has no recognisable `fastq_pass` / `fastq_fail` / `fastq_skip` segment
An `ERROR` event says `skipped symlink — only real files under the output directory are ingested`	The file is a symlink. The watcher does not follow links and the daemon refuses them, so only real files under the validated output directory are read. Copy or move the FASTQ into `fastq_pass/`, or point the stream at the directory the link targets
Nothing is wrong, but the folder is named after an alias	Map the alias itself as the key — the folder name is what matters
A non-barcoded run links nothing	Map the single key `no_barcode`

The mapping cannot be edited while the stream is receiving. Stop it and start a new stream on the same directory with the corrected map; the new watcher re-scans from the beginning, so previously unmapped files are picked up.

The stream will not stop

A run sits in STOPPING until the daemon converts it — that transition belongs to the daemon because the watcher lives in its process. If the daemon was killed between your click and its next tick, the row stays STOPPING forever.

Start the daemon again: on its next tick it clears every STOPPING run, including old ones, and records RUN_STOPPED. In the meantime the Stream page still shows the run, badged Stopping and still polling, because files can land right up until the watcher is torn down. Only the conflict check treats it as finished: a STOPPING run is not ACTIVE, so it does not block a new stream on the same directory.

Counts do not add up

Symptom	Explanation
By barcode totals exceed the header’s totals	Known issue. The header reads the run’s own counters, which correctly skip duplicates; the per-barcode aggregate sums every `FILE_INGESTED` event including those marked `duplicate: true`. A watcher that re-emits a file inflates the per-barcode reads, bases and file count but not the header
An unmapped barcode shows files and size but no reads	By design — unmapped files are never parsed
A file is counted as zero reads	The FASTQ could not be parsed; a line count that is not a multiple of four is treated as unparsable. Often accompanied by `file did not stabilize within timeout — parsing anyway`
A sample shows far more streamed reads than analysable data	Expected. Only the first chunk per sample becomes a `Read`; the rest exist only in the stream ledger

Diagnostics

The daemon’s log. Settings → Background Workers → Show log tails logs/stream-monitor-<pid>.log. Watcher attach and detach, pause and resume transitions, per-file ingest failures and the ingest queue depth are all there.

Database state. On the SeqDesk host:


npx tsx scripts/diagnose-stream-state.ts

It dumps the last five stream-monitor worker rows with their status and age (which is how you spot a zombie row), the five most recent stream runs whatever their status, the latest FILE_INGESTED events with their parsed reads, bases and duplicate flag, and the ledger row count for the newest run.

Spawn diagnostics. If the daemon refuses to start from the interface and the error is not obvious, there is a facility-admin-only endpoint that attempts the spawn and reports everything about the attempt — whether tsx and the script exist, the resolved PATH, the PID, any synchronous throw or asynchronous error event, the exit code, and the first 800 characters of stdout and stderr. It starts the process, watches it for 1.5 seconds, then terminates it. There is no button for it; call it directly:


curl -X POST https://<your-seqdesk>/api/admin/workers/stream-monitor/diagnose

Things that do not work yet

Save yourself the debugging session — none of these is a fault in your setup. Most are simply unbuilt on a feature still under development, so check the known limitations after an update before concluding anything here still holds:

Live device or pore status from MinKNOW. The gRPC settings are stored and probed but never used.
Turning a whole streamed run into one analysable read set.
Running two stream-monitor processes for redundancy. There is no leasing yet; both would attach to every active run.
The Enable stream monitor switch stopping ingest — use Pause on Background Workers instead. See the note on Administration → MinKNOW Stream.

Two more will not change, because they are deliberate rather than unfinished: fastq_fail/ and fastq_skip/ are not ingested, and non-MinKNOW directory layouts are not supported at all — Illumina output belongs in Discover & Associate.