Pipeline Runtime

Pipeline Runtime decides where a Nextflow workflow runs and what environment it runs in. It is the second required piece of infrastructure after Data Storage: without a usable run directory no pipeline can start, and without Conda most packages cannot resolve their tools.

The screen is Pipeline Runtime (/admin/pipeline-runtime). It has no sidebar entry; reach it from Settings → Infrastructure, or from the Pipeline Runtime button on Data Storage. It is split into Required Configuration (always visible) and Advanced Configuration (behind a Show button).

Choosing local or SLURM

SLURM is entirely optional. sbatch, squeue and sacct are only invoked for runs that resolve to SLURM; a purely local instance never needs them installed.

The common production shape is a local global default with SLURM for the heavy packages, set through per-pipeline overrides rather than by flipping the whole instance.

How the target for one run is decided

SeqDesk resolves the execution target per run, in this order — the first match wins:

Each run records the resolved mode, the source of that decision (run, pipeline or global) and a non-secret snapshot of the scheduler settings, so you can still explain an old run after the global settings have changed.

Required Configuration

Queue/Partition only appears when Use SLURM is on.

The test buttons

Each required field has a Test button that probes the live host through POST /api/admin/settings/pipelines/test-setting. They run against the value currently in the box, so you can validate before saving.

The Conda test also reports whether the defaults channel is usable, warning “defaults channel not usable without ToS” when Anaconda’s terms have not been accepted for that installation. Pipelines that pull from conda-forge and bioconda are unaffected.

Advanced Configuration

SLURM resources

Shown only when Use SLURM is on.

Additional SLURM Options is appended verbatim to the submission, so it is the right place for --account=…, --constraint=…, or a QOS flag.

Nextflow Profile Override

pipelines.execution.nextflowProfile, empty by default. When set, it replaces the profile SeqDesk would otherwise choose (for example slurm,conda). Leave it empty unless a package needs a profile the default logic does not produce.

There is no Nextflow path setting. The nextflow binary is resolved from the runtime PATH, or from inside the configured Conda environment.

Per-Pipeline Defaults

A table with one row per installed pipeline package:

Setting the target to Local disables the SLURM columns for that row. The table is empty with “No pipeline packages are installed.” until you install packages on Settings → Pipelines.

The same policy expressed in settings.json:

{
  "pipelines": {
    "enabled": true,
    "execution": {
      "mode": "local",
      "runDirectory": "/data/pipeline_runs",
      "conda": {
        "path": "/opt/conda",
        "environment": "seqdesk-pipelines",
        "cacheDir": "/net/shared/nextflow-conda"
      },
      "slurm": {
        "enabled": true,
        "queue": "cpu",
        "cores": 8,
        "memory": "64GB",
        "timeLimit": 12
      },
      "pipelineOverrides": {
        "mag": {
          "mode": "slurm",
          "slurm": { "queue": "long", "cores": 16, "memory": "128GB", "timeLimit": 48 }
        },
        "metaxpath": {
          "mode": "slurm",
          "slurm": { "queue": "long", "cores": 16, "memory": "128GB", "timeLimit": 48 }
        }
      }
    }
  }
}

Small per-order utilities such as simulate-reads, fastq-checksum and fastqc are normally left to inherit local.

Weblog

Nextflow POSTs run and task events to this URL as they happen, which is what drives live progress and the DAG view. The Test button probes the URL from the app host.

Conda cache directory

pipelines.execution.conda.cacheDir (SEQDESK_CONDA_CACHE_DIR) points Nextflow’s per-process Conda environments at a shared directory, keyed by content hash. On a cluster where compute nodes have no internet access, build the environments once from a host that does — for example by running the pipeline locally first — and the SLURM nodes reuse them from the shared path.

Infrastructure readiness

The sidebar’s Infrastructure entry carries a badge driven by GET /api/admin/infrastructure/readiness:

A red ! means at least one required item is missing; an amber triangle means only recommended items are pending. Clicking the badge jumps to the first gap. The same summary is rendered inline at the top of both Data Storage and Pipeline Runtime.

The Infrastructure page itself (/admin/data-compute) is a hub: two cards linking to the editable screens, plus an Import settings.json panel that can Load example, upload or paste a configuration, Validate JSON as a dry run, and then apply it. It accepts flat keys such as dataBasePath as well as nested ones such as site.dataBasePath, which makes it convenient for cloning a known-good setup onto a second instance.

Verifying the runtime end to end

On a dev or production-like Linux server, the command-line smoke test exercises the whole path over HTTP and the filesystem — no browser needed.

SEQDESK_RUNTIME_E2E_BASE_URL="https://your-seqdesk.example.org" \
SEQDESK_RUNTIME_E2E_EMAIL="admin@example.com" \
SEQDESK_RUNTIME_E2E_PASSWORD="admin-password" \
npm run pipeline:e2e:runtime -- --ensure-dummy-data

Useful variants:

# Local-only check on a host without SLURM
npm run pipeline:e2e:runtime -- --skip-slurm --ensure-dummy-data
 
# SLURM-only check
npm run pipeline:e2e:runtime -- --skip-local --ensure-dummy-data
 
# Assert the configured default policy for the selected pipeline
npm run pipeline:e2e:runtime -- --include-default-policy --expect-default-mode slurm
 
# Use a specific sequencing order instead of the dummy-data one
npm run pipeline:e2e:runtime -- --order-id <order-id>

For the full local + SLURM run the host must have sbatch, squeue and sacct, and must be able to write the configured run directory.

Troubleshooting

See also

• Running a Pipeline — the launch flow that consumes these settings
• Monitoring & DAG Visualization — what the weblog and the monitor worker do
• Data Storage — the other required infrastructure setting
• Configuration Sources — which layer wins