Stream Mode
Stream mode is beta and under active development. It is shipped and it works, and these pages describe precisely what the code does today — but it has had far less real-world use than the rest of SeqDesk, and you should expect bugs. The ones we know about are written down: read Known limitations in this release before you rely on stream mode for anything. That list is a snapshot of the current state, not a contract — most entries are unfinished work rather than deliberate design, they are being worked on, and behaviour, settings and the interface may change between releases. Re-read it after an update.
Stream mode watches the folder MinKNOW is writing into and pulls each new FASTQ into a sequencing order while the flow cell is still running. Instead of waiting for the run to finish, copying files onto the SeqDesk host and matching them to samples with Discover & Associate, you point SeqDesk at the run directory, say which barcode belongs to which sample, and watch reads accumulate per sample.
It is additive. Nothing about the normal path changes — files that arrive from collaborators, or from runs nobody streamed, are still discovered and assigned the usual way.
Three moving parts
Stream mode is the only part of SeqDesk that needs a process outside the web application to do its actual work. Knowing which piece does what explains most of the failure modes.
| Part | What it does | Where it runs |
|---|---|---|
| MinKNOW | Writes basecalled FASTQ files into fastq_pass/ as the run progresses | The sequencing workstation |
The stream-monitor daemon | Watches those folders, parses each file, writes the ledger and the events | A long-lived process on a host that can read the MinKNOW output |
| The SeqDesk web app | Starts and stops streams, stores the barcode map, renders the live view | Wherever SeqDesk normally runs |
The web app never touches the filesystem watcher, and the daemon never serves a page. They communicate only through the database. That is why nothing is ingested when the daemon is not running, no matter how correct the settings look, and why stopping a stream is a request rather than an instant action.
What it is good for, and what it is not
| Designed for | Oxford Nanopore runs driven by MinKNOW, barcoded or not |
| Ingest tier | fastq_pass/ only — fastq_fail/ and fastq_skip/ are never read |
| Granularity | One FASTQ chunk at a time; each chunk is counted and logged |
| Matching | Barcode folder name → sample, mapped by hand before the stream starts |
| Who can use it | Facility admins. The public demo can look but not start or stop |
| Out of scope | Illumina or any non-MinKNOW layout — those belong in Discover & Associate |
| Not built yet | Live gRPC status from MinKNOW; concatenating chunks into one analysable read set |
The single most important limitation: a streamed sample gets one Read
row pointing at its first chunk. Every later chunk is recorded in the
ledger and shown in the interface, but it is not attached to the sample as an
analysable file. Do not launch a pipeline against a streamed sample and assume
it saw the whole run — see From a file on disk to a
sample.
Where to start
- Setting this up for the first time? Setup covers the output root, the path-safety rule that decides whether a directory is accepted, and how to start the daemon.
- No sequencer to hand? The same page explains the built-in simulator, which drips fake FASTQs into a watched folder so you can exercise the whole flow.
- Run in progress? Running a Stream is the operator’s page: pointing at the run folder, mapping barcodes, reading the live surfaces, stopping cleanly.
Related sections
- Administration → MinKNOW Stream — the settings screen, control by control.
- Administration → Background Workers
— starting, pausing and inspecting the
stream-monitordaemon. - File Discovery & Auto-Detect — the non-live path, and what you fall back to when a stream misses something.
- Sequencing Orders & Samples — streams attach to a sequencing order and its samples.
- Reference → Data Model and Reference → App API — the exact fields and endpoints.