Stream Mode Setup

Stream mode is beta and under active development. Setting it up will not disturb anything else — no other part of SeqDesk reads these settings — but expect bugs, and expect the settings on this page to gain fields, change defaults or move between releases. The rough edges we know about are listed under Known limitations in this release.

Two things must be true before a single file is ingested: the MinKNOW output root has to be configured, and the stream-monitor daemon has to be running on a host that can read it. The settings screen alone does nothing.

Before you start

Requirement	Why
FACILITY_ADMIN role	Every stream endpoint and the settings screen reject other roles
Not a demo session	Starting and stopping streams is blocked in the public demo; reading is allowed
Filesystem access to MinKNOW’s output	The daemon reads the files directly — same machine, or an NFS/SMB mount
Read permission for the daemon’s user	The web app’s Test connection and the daemon may run as different users

Configuring the output root

Open the settings screen

Settings → MinKNOW Stream (/admin/minknow-stream). The header shows one of three states for the daemon: ingesting, running, ingestion paused, or daemon stopped.

Set the MinKNOW output root

Under Filesystem watcher, fill in MinKNOW output root. This is the parent directory MinKNOW writes runs into — not a specific run folder. The screen suggests the platform defaults:

Platform	Typical output root
Linux	`/data`
macOS	`/Library/MinKNOW/data`
Windows	`C:\data`

Every stream anyone starts, in any sequencing order, must live under this one path. There is a single global root; it is not per department or per order.

Press Test connection

Test connection runs three read-only checks. Only one of them matters today — Output directory, which proves the path exists and is readable. Judge it on its own: the overall verdict can be carried by the two optional gRPC checks, and it runs as the web application’s user rather than the daemon’s. The full semantics are on Administration → MinKNOW Stream.

Save settings

Save settings writes the configuration. The daemon reads the tunables when it attaches a watcher, so a change takes effect for streams started afterwards.

Every control on the screen — including the Enable stream monitor switch, which despite its label does not stop ingest — is documented field by field under Administration → MinKNOW Stream.

The two settings with no field

Two tunables exist in the stored configuration but have no control on the screen. They matter on network shares, and today they can only be changed by writing the configuration directly:

Key	Default	Purpose
`usePolling`	`false`	Poll instead of using native filesystem events. Needed when MinKNOW writes to an NFS or SMB mount where inotify and FSEvents are unreliable. Costs CPU and I/O on every tick
`stabilityThresholdMs`	`2000`	How long a file’s size must hold steady before the watcher emits it. Raise it on slow shares; it is clamped to a 500 ms minimum

Why a directory gets rejected

Every start request names a directory, and the app checks it against the configured root before creating anything. The rule, in order:

The output root must be configured, and must be an absolute path.
The requested directory must be non-empty and an absolute path.
Both are resolved with realpath — which means every symlink in either path is followed and every .. collapsed. The directory must already exist and be readable; a path that does not resolve is rejected with the underlying filesystem error.
The resolved directory must be inside the resolved root, or be the root itself.

The containment test compares the two canonical paths, so neither a .. segment nor a symlink pointing outside the root can slip through: the symlink is resolved first, and what is compared is where it actually lands. The resolved path — not the string you typed — is what gets stored and watched.

The practical consequences an operator needs:

A run folder that MinKNOW has not created yet is rejected. The directory must exist at the moment you press Start receiving.
A symlink into your MinKNOW data is fine, as long as its target is under the root. A symlink under the root pointing out of it is rejected, and the error names the resolved path so you can see where it landed.
Changing the root does not re-check running streams. They keep watching the directory they resolved to at start time.

The check is a start-time check

Containment is evaluated once, when the run is created — not again on every file. What holds the boundary for the rest of the run is the watcher: it does not follow symlinks.

The watcher does not descend into symlinked directories. A link dropped into fastq_pass/ after the stream started is not walked, so its target is never scanned.
A symlinked FASTQ is refused, not ingested. The link is still reported to the daemon, and the daemon skips it and records an ERROR event reading skipped symlink — only real files under the output directory are ingested. Only real files under the validated directory are read.
What is not re-checked is the directory itself. If the resolved run folder is replaced or re-pointed while the stream is active, nothing revisits the containment decision made at start time. Stop the stream and start it again after any change to the tree above fastq_pass/.

Running the daemon

Nothing is ingested unless stream-monitor is running. There are two ways to start it.

From the interface. Background Workers is the control panel — Start, Stop, Pause, Resume and Show log — and the MinKNOW Stream screen and an order’s Stream view both offer a Start shortcut for the same worker. Pause is the control to reach for when you want ingest to stop without killing the process; it is soft, stored in the database, and survives a restart.

From a shell, on the host that can read the MinKNOW output:


npm run stream:monitor

Under a process manager, run the same script; it handles SIGINT and SIGTERM by detaching every watcher and disconnecting cleanly. Whichever way it is started, it must run somewhere that can read the MinKNOW output — the same machine as MinKNOW, or a host with the share mounted. It does not have to be the machine serving SeqDesk, as long as both reach the same database.

Environment variables the daemon honours:

Variable	Default	Effect
`INGEST_CONCURRENCY`	`4`	Simultaneous ingest jobs across all streams. Fits under the default database pool of 10
`STREAM_MONITOR_INTERVAL_MS`	the Poll interval setting	Overrides the reconcile interval

There is also a --once flag, which reconciles once, waits two seconds for the watcher’s initial scan, and exits. It is useful for a scripted check that ingest works at all.

Trying it without a sequencer

The same script can play the part of the sequencer. It writes small FASTQ files into <output dir>/fastq_pass/<barcode>/ on a timer, named the way MinKNOW names them, so the whole path — watcher, parser, ledger, live view — runs against real files.

The one-click route. On the MinKNOW Stream screen, in development builds, Test without a real MinION → Use simulator preset does everything: it points the output root at /tmp/seqdesk-sim, saves, makes sure stream-monitor is running, and starts the stream-simulator worker. Then open any sequencing order’s Stream view — in development the output directory field is pre-filled with /tmp/seqdesk-sim — map barcode01, barcode02 and barcode03 to samples, and press Start receiving. Files appear every 15 seconds.

From a shell:


npm run stream:monitor:simulate

The npm script writes to /tmp/fake-minknow, while the simulator preset and the pre-filled form field both use /tmp/seqdesk-sim. If you start the simulator from the shell, pass the directory you actually configured: npx tsx scripts/stream-monitor.ts --simulate --output-dir=/tmp/seqdesk-sim.

Variable	Default from the shell	Value the preset uses
`--output-dir=`	`/tmp/fake-minknow`	`/tmp/seqdesk-sim`
`SIMULATE_INTERVAL_MS`	`5000`	`15000`
`SIMULATE_BARCODES`	`barcode01,barcode02,barcode03`	the same three

Each generated file holds 50 reads of 200 bases. Start the simulator before you start the stream — the run directory has to exist for the start request to be accepted, and the simulator is what creates it.

The simulator workers are marked dev-only: they are hidden from Background Workers in production builds, and an attempt to start one there is refused with 403.