Pipeline Runtime

Pipeline runtime settings control how Nextflow workflows are executed — locally or on a SLURM cluster — and which tools are available.

Configuration Location

Pipeline runtime settings are at Admin → Pipeline Runtime (/admin/pipeline-runtime), or via the config file / environment variables.

Execution Mode

The global default maps to the SEQDESK_PIPELINE_MODE environment variable (config key pipelines.execution.mode, values local or slurm).

SeqDesk resolves the execution target for every run in this order:

1. Run override selected by a facility admin when starting the run
2. Per-pipeline override from Admin → Pipeline Runtime
3. Global runtime setting
4. Local default

This means SLURM is optional. Keep the global mode local for small utilities, then set heavier pipelines such as MAG or MetaxPath to SLURM.

Local Execution

Nextflow runs as a child process on the SeqDesk server. All compute happens on the same machine.

SLURM Execution

Nextflow submits individual processes as SLURM jobs. Configure:

SLURM commands are only required for runs that resolve to slurm. Local runs do not call sbatch, squeue, or sacct.

Per-Pipeline Overrides

Use Admin → Pipeline Runtime → Pipeline overrides to choose a default target per installed pipeline. The common production pattern is:

Infrastructure JSON can express this policy:

{
  "pipelines": {
    "enabled": true,
    "execution": {
      "mode": "local",
      "runDirectory": "/data/pipeline_runs",
      "slurm": {
        "enabled": true,
        "queue": "cpu",
        "cores": 8,
        "memory": "64GB",
        "timeLimit": 12
      },
      "pipelineOverrides": {
        "mag": {
          "mode": "slurm",
          "slurm": {
            "queue": "long",
            "cores": 16,
            "memory": "128GB",
            "timeLimit": 48
          }
        },
        "metaxpath": {
          "mode": "slurm",
          "slurm": {
            "queue": "long",
            "cores": 16,
            "memory": "128GB",
            "timeLimit": 48
          }
        }
      }
    }
  }
}

Admins can still override the target for a single run from the launch dialog. Each run records the resolved execution mode and a non-secret scheduler snapshot so later debugging does not depend on changed global settings.

Conda Configuration

Nextflow pipelines use Conda for dependency management:

The Conda environment must have Nextflow installed or be configured to find it.

Nextflow Settings

The Nextflow binary is resolved automatically — there is no “Nextflow path” admin field. Nextflow must be available on the runtime PATH (or inside the configured Conda environment).

Additional Nextflow configuration available in the admin UI:

• Nextflow profile — profile name to use (e.g., conda, docker)

Reference-database download paths (GTDB-Tk and other pipeline databases) are not configured here. They are managed on the Pipelines page (/admin/settings/pipelines), where each pipeline’s databases can be downloaded and their status reviewed. The download directory can also be set with the SEQDESK_PIPELINE_DATABASE_DIR environment variable (config key pipelines.databaseDirectory).

Weblog Setup

Real-time pipeline monitoring requires configuring the Nextflow weblog:

The weblog URL is automatically included in the Nextflow execution command. See Monitoring & DAG Visualization for details.

Pipeline Run Directory

Each pipeline run creates a subdirectory (e.g., MAG-20240126-001/) containing the samplesheet, scripts, logs, and outputs.

Infrastructure Validation

The Admin → Infrastructure page (/admin/data-compute) validates that your runtime environment is correctly configured before you run any pipelines. It tests four areas:

Each check shows a pass/fail indicator. Fix any failures before attempting to run a pipeline.

Import Setup JSON

The infrastructure page also supports importing a complete infrastructure configuration as JSON. You can:

• Upload a JSON file or paste a configuration
• Load an example to see the expected format
• Validate (dry-run) before saving
• Save to apply the configuration

This is useful for automating setup or copying configurations between instances. The JSON supports both flat keys (e.g., dataBasePath) and nested keys (e.g., site.dataBasePath), as well as alternate names for flexibility.

Headless Runtime Smoke Test

On a dev or production-like Linux server, run the command-line smoke test from the SeqDesk application directory. It uses only HTTP/API calls and filesystem checks; no browser is required.

SEQDESK_RUNTIME_E2E_BASE_URL="https://your-seqdesk.example.org" \
SEQDESK_RUNTIME_E2E_EMAIL="admin@example.com" \
SEQDESK_RUNTIME_E2E_PASSWORD="admin-password" \
npm run pipeline:e2e:runtime -- --ensure-dummy-data

The default run:

Useful variants:

# Local-only check on a host without SLURM
npm run pipeline:e2e:runtime -- --skip-slurm --ensure-dummy-data
 
# SLURM-only check
npm run pipeline:e2e:runtime -- --skip-local --ensure-dummy-data
 
# Check the configured default policy for the selected pipeline
npm run pipeline:e2e:runtime -- --include-default-policy --expect-default-mode slurm
 
# Use a specific order instead of the dummy-data order
npm run pipeline:e2e:runtime -- --order-id <order-id>

For a full local + SLURM check, the host running the command must have sbatch, squeue, and sacct available and must be able to write the configured pipeline run directory.