Skip to Content
Pipelines & AnalysisOverview

Pipelines & Analysis

A pipeline in SeqDesk is a packaged Nextflow workflow plus a contract that says what SeqDesk should feed it and what SeqDesk should do with what comes back. The workflow itself is ordinary Nextflow. Everything that makes it part of SeqDesk — which samples it may run on, what samplesheet to generate, which produced files map to which sample, and which database columns may be updated — lives in four declarative files inside the package directory.

That single idea explains the whole section. SeqDesk is not a workflow manager competing with Nextflow; it is the layer that turns “a workflow ran” into “these samples now have assemblies, these reads now have checksums, and this report is visible to the researcher who ordered the sequencing”.

The mental model

pipelines/<id>/                 the package (manifest, registry, definition, samplesheet)
      │
      ▼  discovered at startup, one catalog entry per package
Catalog entry                   study-scoped, order-scoped, or both
      │
      ▼  a FACILITY_ADMIN picks samples and presses Start Pipeline
PipelineRun  (status pending → queued → running → completed | failed | cancelled)
      ├── generated samplesheet.csv      built from canonical Sample/Read rows
      ├── run.sh                         local `bash run.sh`, or `sbatch run.sh`
      └── output/                        everything Nextflow writes
      │
      ▼  on completion: discovery matches files → declared outputs
PipelineArtifact / Assembly / Bin / Read writeback
      │
      ▼  optionally: Use as final
Visible to the researcher who owns the study or sequencing order

Five facts follow from that diagram, and they cause most of the questions:

  • Only a FACILITY_ADMIN can create, start, cancel or delete a run. Every mutating pipeline route checks the role. Researchers never launch anything.
  • A run always has exactly one target — one study or one sequencing order, never both, never several. POST /api/pipelines/runs rejects the request with Exactly one of studyId or orderId is required otherwise.
  • A researcher sees nothing until a run is published. Even the owner of the study or order gets 403 Forbidden on a run’s API routes until an admin marks it Use as final. See Results.
  • Database writes are made by SeqDesk, not by the workflow. A package declares which Read fields it may touch; the write itself happens centrally after the declared contract is validated. A workflow can never write directly into SeqDesk’s database.
  • Nothing is overwritten silently. Raw reads are protected, cleaned reads are staged for review, and a superseded read record is deactivated rather than deleted.

Where to start

  • You want to see what this installation supports or add a workflow. Start with Installing & Setting Up Pipelines. It covers the command-line catalog, order and study filters, package installation, runtime setup, readiness, databases, and a safe Simulate Reads walkthrough.
  • You want to run something today. Read Study vs Sequencing Order Pipelines to pick the right surface, then Running a Pipeline. If the install is new, start with Study Demo Report — it is deterministic, needs no reference data, and proves execution, monitoring, output discovery and the results UI in one go.
  • A run is stuck, or shows a status you do not expect. Go to Monitoring a Run. Most surprising statuses come from one of two places: the Nextflow weblog cannot reach the app, or the scheduler and the trace file disagree.
  • A run finished but nothing appeared in SeqDesk. Go to Results & Outputs. Output discovery is glob-based and per-package; if a pattern does not match, the run still completes and the artifact is simply absent.
  • You want to add a workflow. Adding Custom Pipelines is the long-form reference, built around the smallest real package in the repository.

Prerequisite. Pipelines are a facility feature and need a working execution environment — Nextflow, Java, and Conda — plus a writable run directory. seqdesk pipelines install <name> checks those requirements and offers guided runtime setup; you can also configure an existing environment in Administration → Pipeline Runtime. A package is enabled only after its full readiness check passes.

Installing & Setting Up PipelinesDiscover order and study workflows, install a package, finish its runtime, configuration, database and storage setup, and verify readiness.Study vs Sequencing Order PipelinesThe two scopes, where each one is launched from, what it may write back, and how to choose.Running a PipelinePrerequisites, the Analysis surface, sample selection, execution target, and the exact errors a refused launch produces.Monitoring a RunRun states, the weblog, the trace fallback, scheduler reconciliation, and what “Finalizing…” means.Available PipelinesEvery package that ships with SeqDesk, its scope, parameters, outputs and writeback.Adding Custom PipelinesPackage layout, manifest contract, samplesheet generation, output discovery, validation and store submission.Results & OutputsArtifacts, assemblies, bins, read writeback, candidate review, publishing runs and deleting them.