File Discovery & Auto-Detect

Discovery is a read-only walk of one configured directory. It never opens a FASTQ file, never moves anything, and never writes to disk. It produces a list of candidate files, which the matcher then turns into per-sample suggestions.

Run it from the Associate view of an order’s Sequencing Data page (/orders/[id]/sequencing?view=discover) with the Auto-Discover button.

What the scanner does

In order:

1. Resolve the base path. A missing directory raises Directory does not exist: <path>; a file where a directory was expected raises <path> is not a directory.
2. Walk to scanDepth levels. Depth 1 is the base directory itself.
3. Skip anything matching an ignore pattern — checked against the path relative to the base, for directories as well as files.
4. Keep files whose name ends with an allowed extension (case-insensitive).
5. Skip files modified within activeWriteMinAgeMs so a sequencer that is still writing is not picked up mid-file.
6. Stop at 10,000 matching files and flag the result as truncated.
7. Sort by filename and cache the result.

Directories that cannot be read are skipped rather than aborting the scan, and each one is reported in the scan warnings with its error message.

Settings

The scanner reads its settings from SiteSettings.extraSettings.sequencingFiles. Most of them are editable under Administration → Data Storage; the seqdesk.config.json file exposes the same section.

In the configuration file the extension list can be written as extensions or allowedExtensions; the runtime scanner reads allowedExtensions, which is what the admin settings API writes.

Scan depth is the setting that bites first

scanDepth counts the base directory as level 1.

A typical Oxford Nanopore output tree looks like <base>/20260415/fastq_pass/barcode07/FAX12345_pass_barcode07_a1b2c3d4_0.fastq.gz — four levels down. With the default scanDepth of 2 the scan returns nothing and the discovery summary reports every sample as no match.

Two ways out, and the first is usually better:

• Point the base path deeper. Set site.dataBasePath to the run directory so the barcode folders sit one level below it.
• Raise scanDepth. Effective, but the scan gets slower and much more likely to hit the 10,000-file ceiling on a shared filesystem.

The in-flight write guard

A file whose modification time is newer than activeWriteMinAgeMs (30 seconds by default) is skipped, and its relative path and timestamp are recorded in the scan warnings — up to the first 25. The Scan completed with warnings banner shows the count and the first three paths.

This exists so that basecalling in progress does not get linked as a finished read. If a scan looks incomplete immediately after a run, wait half a minute and press Auto-Discover again.

Set activeWriteMinAgeMs to 0 to disable the guard entirely — only sensible when the data directory is a static archive.

Supported file naming

Pairing works by stripping known suffixes from the filename to recover a base identifier, then grouping files that share one.

Stripped, in this order: the extension (.fastq.gz, .fq.gz, .fastq, .fq); a trailing Illumina chunk suffix of the form _00N; a read marker _R1, _R2, .R1, .R2, _1, _2, .1, .2; a lane marker of the form _L001; a sample index of the form _S1.

Pairing logic

Files are grouped by identifier, then:

• both an R1 and an R2 present → paired-end;
• only one file → single-end;
• a file that matches neither read marker → treated as R1, that is, single-end;
• two files claiming the same role for the same identifier → the first one in the scan’s alphabetical order wins and the other is dropped from that pair.

That last case is the one to watch: a directory holding sample_R1_001.fastq.gz and sample_R1_002.fastq.gz collapses both to the identifier sample and only one survives the pairing. Concatenate multi-chunk Nanopore or NovaSeq output before discovery, or assign it manually.

The scan cache

Results are cached in memory for 5 minutes, keyed by the resolved base path and every scan option — extensions, depth, ignore patterns, file limit and the active-write threshold. Change any setting and the next scan is fresh automatically.

The Auto-Discover button always sends force: true, so pressing it bypasses the cache and re-walks the directory. The cache only affects programmatic callers that omit the flag.

Reading the scan warnings

Browsing instead of scanning

Manual association does not use the discovery scan. The file picker calls a separate browse endpoint that walks scanDepth + 3 levels and returns up to 250 files (500 from the association view), filtered by a search string and annotated with each file’s current assignment.

That deeper walk is deliberate: when discovery has failed because the tree is deeper than scanDepth, the browser can still reach the files so you can link them by hand.

Worked example

A Nanopore run with barcodes, done properly:

1. Set the data base path to the run directory, so it contains fastq_pass/barcode01/….
2. Set scanDepth to 3 — base, fastq_pass, barcodeNN.
3. Add **/fastq_fail/** to the ignore patterns so failed reads are never suggested.
4. Create a run plan so each sample carries its barcode.
5. Press Auto-Discover. Files under barcode01/ match the sample whose run-plan barcode normalises to barcode01, at confidence 0.92 (single-end).
6. Review, then Apply n Exact Matches.

Troubleshooting

Next

• Assigning Files to Samples — turning discovered files into Read records.
• Sequencing Runs — the barcode plan that makes matching deterministic.