Running a Pipeline

Starting a pipeline is a two-phase operation, and knowing that explains most of what you see. First SeqDesk creates a run — validating metadata, inputs and config, then storing a pending row. Then it starts the run — generating the samplesheet, the Nextflow config and run.sh, and handing the script to bash or sbatch. The UI does both when you press Start Pipeline, but they fail differently, and the error messages tell you which phase refused.

Before you start

Requirement	Why	Where
FACILITY_ADMIN role	every mutating pipeline route rejects other roles with `403`	—
Not a demo session	pipeline execution is disabled in the public demo (`Pipeline execution is disabled in the public demo.`)	—
The pipeline is enabled	a disabled package returns `Pipeline <id> is disabled` (403)	Admin → Pipelines
Nextflow, Java, and Conda installed	the generated `run.sh` invokes them directly	Pipeline Runtime
A writable run directory	every run gets its own folder under it; default `/data/pipeline_runs`	Pipeline Runtime
A data base path	read files are resolved relative to it when the samplesheet is built	Data Storage
Reads linked to samples	required by every package except Simulate Reads, Study MultiQC and Study Demo Report	Assigning Files to Samples

A newly installed Store package remains disabled until its runtime, configuration, reference data, and paths pass the readiness check; the installer enables it automatically when they do. Legacy bundled packages with no explicit PipelineConfig row are enabled by default unless the installation profile set an allowlist. See Installing & Setting Up Pipelines.

Launching a study pipeline

Open the study’s Analysis surface

Go to the study and choose Analysis in the sidebar (/studies/{id}?tab=pipelines). You get an overview card per enabled pipeline, each showing whether it has runs and what state the newest one is in (Running, Completed, Failed, or Not run yet).

Submission-category pipelines are deliberately not here — Submit to ENA lives under Publishing. Everything else appears under Analysis.

Select the pipeline

Click a card. You land on the per-pipeline page: a header with the package name and description, a Start Pipeline button, an Execution Target control, a Settings block, a Samples table, and the Pipeline Runs table for this study and pipeline.

If the header shows an amber button instead of Start Pipeline, the environment check failed. The button text is the reason (missing Nextflow, an unwritable run directory, and so on); click it to re-check.

Check the Samples table

The table lists the samples that are eligible for this pipeline, with their read files, source, and any blocking issues. Ineligible samples are surfaced with a reason rather than silently dropped, so an unexpectedly small eligible count is usually a reads problem, not a pipeline problem.

Adjust Settings

Settings renders the package’s configSchema. Fields are split by their x-seqdesk.placement:

basic — always visible.
advanced — hidden behind Advanced settings.
derived — shown read-only, because SeqDesk computed the value from the data (for example the sequencing platform).
admin / hidden — not editable per run at all; these are facility-level settings, typically reference database paths pinned by the install profile.

Anything you leave alone uses the package’s defaultConfig. The values you send are snapshotted onto the run (PipelineRun.config), so a later change to the defaults does not rewrite history.

Pick an execution target

The Execution Target control offers three choices: Default (Local) or Default (SLURM) — whichever the resolved policy says — plus explicit Local and SLURM. The SLURM option is disabled, with the reason in its tooltip, when the availability probe cannot find a working scheduler on the host.

The resolved default comes from the global execution settings, optionally overridden per pipeline. The choice you make is stored on the run as executionMode, and a snapshot of the non-secret policy is stored in executionProfile, so a run always records how it was meant to execute.

Start Pipeline

SeqDesk validates, creates the run, then starts it. On success the new run appears at the top of Pipeline Runs with a run number and status queued.

Launching a sequencing order pipeline

Open the order’s Analysis surface

Go to the sequencing order and choose Analysis in the sidebar (/orders/{id}/sequencing?view=analysis). As on a study, you get one card per enabled order pipeline; clicking one opens its per-pipeline page.

Review readiness per sample

The order surface is per-sample. Each sample row shows whether it is ready and, if not, why — Missing reads, Missing R2 file, Needs raw or unknown reads, or Files missing. If the order has no reads at all yet, start with Simulate Reads, which generates them.

Configure

The Settings block works exactly as on the study side. Order pipelines are usually narrower: FASTQ Checksum, FastQC and NanoPlot have no parameters at all, while Simulate Reads and Read Cleaning have a handful.

Run one sample, or Run All Ready

Each ready sample has its own run button; the header has Run All Ready with a count badge. Each launch is a separate PipelineRun — one broken sample does not stop the others.

What “Start” actually does

The start route claims the run atomically and refuses anything that is not pending:


Cannot start run with status: running    → 400

That is the double-start guard. Two clicks, or a click plus a retry, cannot produce two executions of the same run.

Preparation then, in order:

Resolves execution settings and the effective policy for this pipeline.
Generates the samplesheet from the canonical Sample and Read rows.
Allocates the run number {PIPELINE}-{YYYYMMDD}-{NNN} — for example MAG-20260415-003. The counter is the highest existing number for that pipeline that day, plus one; a collision between two concurrent prepares is retried up to five times.
Creates the run folder named after the run number, writes samplesheet.csv, nextflow.config and run.sh, and creates output/ and logs/.
Sets runFolder, outputPath, errorPath, status queued, queuedAt.
Executes — locally as a detached bash run.sh (tracked as queueJobId = local-<pid>), or via sbatch --parsable run.sh (tracked as the numeric SLURM job id).

The run number you see immediately after creation is a placeholder of the form MAG-1776123456789-K3P9Q. It is replaced with the canonical MAG-YYYYMMDD-NNN when the run is prepared. A run that never got past validation keeps its placeholder, which is a quick way to spot one.

Input generation

SeqDesk builds the Nextflow input file from the database — never from whatever happens to be on disk. The shape is declared in the package’s samplesheet.yaml, one row per sample, and file columns are prefixed with the configured data base path so the workflow receives absolute paths.

For the MAG Pipeline the generated samplesheet.csv has six columns:

Column	Source	Notes
`sample`	`sample.sampleId`
`group`	`study.id`	nf-core/mag uses it for co-assembly grouping
`short_reads_1`	`read.file1`	paired reads only, path-prefixed, required
`short_reads_2`	`read.file2`	paired reads only, path-prefixed, required
`short_reads_platform`	`order.platform`	mapped to `ILLUMINA` / `DNBSEQ` / `BGISEQ` / `OXFORD` / `PACBIO`
`long_reads`	—	always empty; MAG’s long-read branch is not wired up

short_reads_platform is derived from the order’s sequencing-technology selection, falling back to the legacy Order.platform string. A worked example row:


sample,group,short_reads_1,short_reads_2,short_reads_platform,long_reads
S-1755262041412-K7QX2,cly8v3s0000abcd,/data/seq/ORD-.../S1_R1.fastq.gz,/data/seq/ORD-.../S1_R2.fastq.gz,ILLUMINA,

Simpler packages generate simpler sheets — FastQC emits sample_id,fastq_1,fastq_2, with fastq_2 blank for single-end samples.

Execution modes

Local

run.sh is executed directly on the SeqDesk host as a detached process. Good for testing, demo data, and any facility without a cluster.

Local runs get one extra safety net: SeqDesk caps per-process resource requests to roughly 90% of the host’s memory and its full CPU count via Nextflow’s resourceLimits. Without that, an nf-core process declaring 64 GB would be rejected pre-flight on a smaller machine before it ever ran.

SLURM

run.sh is an sbatch script and is submitted with sbatch --parsable. The generated header uses the configured execution settings:

Setting	Default	Emitted as
Queue	`cpu`	`#SBATCH -p cpu`
Cores	4	`#SBATCH -c 4`
Memory	`64GB`	`#SBATCH --mem='64GB'`
Time limit	12	`#SBATCH -t 12:0:0`
Additional options	—	appended verbatim as a further `#SBATCH` line

The time limit is in hours, not minutes. 12 becomes -t 12:0:0. Setting it to 60 asks for sixty hours, which most partitions will never schedule — the job sits PENDING forever with reason (PartitionTimeLimit).

By default a SLURM run submits one SLURM job per Nextflow process (process.executor = 'slurm'). On clusters that cap concurrent submissions per user, that nested submission fails, so the runtime supports an inline executor mode: the whole run stays inside a single sbatch allocation and Nextflow uses its local executor inside it. That trades cluster-wide parallelism for a single job slot — and it has a consequence worth knowing: resourceLimits is not applied in SLURM mode, so a process declaring more CPUs than the allocation has is rejected pre-flight with Process requirement exceeds available CPUs. If you use inline mode, request enough cores and cap the pipeline’s own thread parameters.

SLURM’s own --output / --error are written to node-local /tmp and copied back into the run folder as logs/slurm-<jobid>.out / .err when the job exits, because the scheduler daemon often cannot write into a root-squashed NFS run directory.

The run directory


{PIPELINE}-{YYYYMMDD}-{NNN}/
├── run.sh              generated launch script
├── samplesheet.csv     generated package input
├── nextflow.config     weblog, executor and report-overwrite settings
├── output/             Nextflow --outdir; everything discovery scans
├── logs/
│   ├── pipeline.out    stdout, and the canonical exit marker
│   ├── pipeline.err    stderr
│   └── slurm-<jobid>.* copied back from the scheduler (SLURM only)
├── trace.txt           per-task trace (TSV)
├── dag.dot             workflow DAG (Graphviz)
├── report.html         Nextflow execution report
├── timeline.html       Nextflow timeline
└── work/               Nextflow work directory

The last line of logs/pipeline.out is the one SeqDesk trusts most:


Pipeline completed with exit code: 0 at Wed Apr 15 11:42:03 CEST 2026

It is written by an EXIT trap, so it appears exactly once and carries the real status even when a command aborts under set -e. Its absence means the wrapper has not finished. See Monitoring a Run.

When a launch is refused

Validation happens before a run row exists, so these produce an error in the UI and no run at all.

Message	What it means
`Exactly one of studyId or orderId is required`	the request targeted both or neither
`Invalid pipeline ID`	no package with that id is installed
`Pipeline <id> is disabled`	the pipeline is switched off in admin settings or excluded by the install profile
`Pipeline <id> does not support <type> targets`	wrong scope — see Pipeline Scopes
`Study not found` / `Sequencing Order not found`	the target was deleted between page load and launch
`Invalid sample IDs: …`	selected samples do not belong to the target
`Pipeline metadata validation failed`	one or more blocking metadata issues; the details list names each one
`Pipeline input validation failed`	the package’s own input check rejected the selection (usually missing or unpaired reads)
`Pipeline config validation failed`	a settings value is outside the schema
`executionMode must be one of: default, local, slurm`	an invalid execution target was requested
`Data base path not configured in settings`	reads cannot be resolved to absolute paths
`Pipeline package not found: <id>`	the catalog entry exists but the package directory does not
`Failed to prepare run`	samplesheet or run-directory preparation failed; the details list carries the cause

Two metadata failures are common enough to name:


Sample <id> is missing paired-end read files
Sequencing technology "MinION (ont-minion)" incompatible with MAG.
MAG requires short-read technologies that support paired reads.

The first means the sample has a Read row but no file2. The second means the order’s sequencing technology is long-read or single-end-only; MAG requires a short-read, paired-capable technology, and no configuration flag overrides that. Submit to ENA adds its own long list — missing study accession, missing taxId, missing checksums, missing assembly — all documented in Preparing a Submission.

Retrying a failed run

A failed run’s detail page shows a Retry button next to Refresh. Retry does not restart the old run: it creates a new run with the same pipeline, the same sample selection and the same config, optionally on a different execution target chosen in Retry Target. The failed run stays in the table as a record.

Retry only works for study runs. A run without an associated study reports Run is missing an associated study. To repeat an order pipeline, launch it again from the order’s Analysis surface — which is a single click per sample anyway.

Study vs Sequencing Order Pipelines — choosing the right surface.
Monitoring a Run — what happens after the launch.
Results & Outputs — where the produced files end up.
Available Pipelines — parameters and outputs per package.
Administration → Pipeline Runtime — configuring executors, conda and the weblog.