How Stream Mode Works

Stream mode is beta and under active development. Everything on this page is what the code does today, and today is all it describes: several parts are marked in the source as first-pass work, and the mechanics below — states, events, what a chunk turns into — can change from one release to the next. Expect bugs; the ones we know about are collected under Known limitations in this release.

The lifecycle

A stream is a StreamRun row bound to one sequencing order and one watched directory. It has three states and no error state.


          POST …/stream                 monitor tick
  (none) ──────────────▶ ACTIVE ─────────────────────────▶ ACTIVE
                            │   (ingesting, heartbeating)
              POST …/stop   │
                            ▼
                        STOPPING ──── monitor tick ────▶ STOPPED

Only the daemon performs the last transition. The API sets STOPPING and records a RUN_STOP_REQUESTED event; on its next tick the daemon closes the watcher, writes STOPPED plus stoppedAt, and records RUN_STOPPED. If the daemon is not running, the row stays STOPPING indefinitely — that is a diagnostic, not a bug in your setup.

Starting is guarded twice. Two ACTIVE streams may never watch the same resolved directory, and the check, the insert and the RUN_STARTED event happen inside one SERIALIZABLE transaction, so two simultaneous starts (or a double-click) cannot both win. The loser gets a 409 naming the run and order that already holds the directory.

What the daemon does on each tick

The stream-monitor process wakes on an interval (the Poll interval setting, default 5000 ms) and reconciles reality against the database:

Reads the pause flag. When the worker is paused, watchers stay attached but every new file is skipped — deliberately, so nothing is marked seen and files get another chance on resume.
Tears down the watcher for every STOPPING run and moves it to STOPPED.
Attaches a watcher for every ACTIVE run it is not already watching, and detaches watchers for runs that are no longer ACTIVE. On attach it stamps monitorId (hostname:pid:random) so you can tell which process is ingesting.
Heartbeats lastSeenAt and heartbeatAt on the runs it owns.

The poll interval only controls how fast newly started and stopped streams are noticed. File pickup itself is event-driven and does not wait for a tick.

The watcher is attached to <run directory>/fastq_pass — not to the run directory as a whole. Files under fastq_fail/ and fastq_skip/ are never ingested, and neither is anything else in the run folder. Symlinks are not followed, so nothing that lives outside the watched directory is reached through one either.

From a file on disk to a sample

When a file appears under fastq_pass/, six things decide what happens to it.

Extension filter

Only .fastq, .fastq.gz, .fq and .fq.gz are considered. Everything else is ignored silently.

Symlink refusal

The watcher is attached with followSymlinks: false, so a symlinked directory inside fastq_pass/ is never descended into. A symlinked file is still reported, and the daemon refuses it explicitly: it records an ERROR event reading skipped symlink — only real files under the output directory are ingested and moves on. Only real files under the validated output directory are read, which keeps ingest inside the boundary the start-time containment check established.

Stability wait

The watcher waits for the file size to stop changing before emitting it — for stabilityThresholdMs, 2000 ms by default — and the daemon then re-checks stability itself, polling until two consecutive sizes agree for 1.5 s and giving up after 15 s. If that wait times out the file is parsed anyway and an ERROR event records that it never stabilised.

Barcode resolution from the path

The folder the file sits in — not the filename — decides the barcode key:

Directory under `fastq_pass/`	Key used for lookup	Notes
`barcode07/`	`barcode07`	`barcode` + 2 to 4 digits, any case
`Barcode007/`	`barcode007`	lower-cased
`BC07/`	`barcode07`	`bc` + digits is accepted and rewritten
`unclassified/`	`unclassified`	reads MinKNOW could not assign
`my_sample_alias/`	`my_sample_alias`	any other folder name is taken as an alias and lower-cased
(no subfolder — files sit directly in `fastq_pass/`)	`no_barcode`	a non-barcoded run

Sample lookup

The key is looked up in the stream’s barcode map. Map keys are lower-cased when you save them, and the resolved key is always lower-case, so case never matters. Three outcomes:

No entry for the key. A FILE_INGESTED event is recorded with linkedSampleId: null and reason: "barcode not mapped to a sample". The file is not parsed, not added to the ledger, and the run totals do not move. It shows up in By barcode as an unmapped row with a file count and zero reads.
Entry points at a sample that is not in this sequencing order. An ERROR event is recorded and the file is skipped.
Entry points at a sample in this order. Ingest proceeds.

Counting and linking

Reads and bases are counted by actually parsing the FASTQ (gzip is handled transparently), not estimated from the byte size. A file whose line count is not a multiple of four is treated as unparsable and counted as zero.

The file is then inserted into the ledger. The unique constraint on (streamRunId, filePath) is the idempotency mechanism: if the row already exists the daemon records a FILE_INGESTED event marked duplicate: true and skips the totals increment, so a watcher that re-emits on a rename, an atomic write or a restart cannot double-count.

What the sample actually gets

This is the part most likely to surprise you.

The first chunk ingested for a sample seeds a Read row with file1 set to that chunk’s path — but only if the sample has no Read row at all yet. Every subsequent chunk is written to the stream ledger and shown in the interface, and is never attached to the sample as a file. The source calls promoting the rest into one concatenated read set a follow-up step; it is not implemented.

Two consequences worth stating plainly:

A pipeline launched against a streamed sample sees one chunk, not the run.
If the sample already had a Read from any other route — an upload, Discover & Associate — the stream creates nothing, and its data stays entirely outside the analysable file set.

The seeded row is created with only sampleId and file1, so it takes the schema defaults: dataClass cleaned, dataClassSource legacy_assumed_cleaned, isActive true. Raw nanopore basecalls are therefore labelled as cleaned data unless you reclassify them.

What gets written

Table	One row per	Purpose
`StreamRun`	Stream session	Status, watched directory, barcode map, running totals, monitor ownership
`StreamIngestedFile`	Ingested file	The ledger — path, barcode, sample, size, reads, bases. Unique on `(streamRunId, filePath)`
`StreamRunEvent`	Anything that happened	Append-only, auto-incrementing `seq`, JSON payload
`Read`	First chunk per sample	The only link into the normal sequencing-data model

Field-by-field detail is in Reference → Data Model.

Event kinds

Kind	Written by	Payload
`RUN_STARTED`	The start endpoint	`outputDir`, `barcodeMap`
`FILE_INGESTED`	The daemon	`filePath`, `barcode`, `size`, `reads`, `bases`, `linkedSampleId`; plus `duplicate: true` on a re-emit, or `reason` when the barcode was not mapped
`RUN_STOP_REQUESTED`	The stop endpoint	`stoppedBy: "user"`
`RUN_STOPPED`	The daemon	`monitorId`
`ERROR`	The daemon	A `message` plus whatever context applies — unparsable path, missing `outputDir`, sample not in this order, a refused symlink, ingest failure

The Audit trail panel in the interface documents four of these; RUN_STOP_REQUESTED is not in its legend but does appear in the feed.

Concurrency and load

All ingest work across every stream on a host passes through one FIFO semaphore, four jobs at a time by default (INGEST_CONCURRENCY). Decompressing and counting several large FASTQs at once would otherwise exhaust the database connection pool and peg the CPU. When the queue backs up, the daemon logs its depth — which is exactly when you want to see it.

Known limitations in this release

Verified against the shipped code, not against the interface copy. Read this as a snapshot of where stream mode currently stands, not as a description of how it is meant to work: most of the entries below are unfinished work on a beta feature and are being addressed, so check the list again after an update. Where something is a deliberate boundary rather than a gap, it says so.

Only the first chunk per sample becomes a Read. There is no finalize or concatenate step yet; the source marks promoting the rest as follow-up work. This is the limitation most likely to affect your results — see what the sample actually gets.
The MinKNOW gRPC connection is not implemented. The MinKNOW host, gRPC port and TLS CA cert path settings are stored and can be probed by Test connection, but nothing reads them at ingest time and the daemon logs gRPC enrichment: not configured in MVP on startup. There is no live pore or device status, and the settings are in place ahead of the feature.
minknowRunId, flowCellId and deviceId are always empty in practice. The start endpoint accepts them, but the Stream screen never sends them.
There is no reclaim of a dead monitor’s runs. monitorId and heartbeatAt are recorded, but nothing acts on a stale heartbeat, and there is no leasing that would stop two monitors attaching to the same run. Run one monitor until that changes.
Barcodes are matched by folder name only. The ONT filename parser exists and understands flow cell, pass/fail/skip, duplex, alias and batch number, but ingest does not use it.
The per-barcode aggregate counts duplicate re-emits. The run’s own totals skip them, so the two figures can disagree — see counts do not add up.
fastq_fail/ and fastq_skip/ are never ingested. Deliberate: the watcher is attached to fastq_pass/ and only the pass tier is treated as data.
The barcode map is fixed for the life of a stream. Deliberate, for the sake of an unambiguous audit trail. To change it, stop the stream and start a new one.