Architecture

SeqDesk is a single Next.js application backed by one PostgreSQL database, plus a small number of helper processes and an external workflow engine. It is designed to be self-hosted on one machine, optionally submitting compute to a SLURM cluster that shares a filesystem with it.

Understanding two things explains most operational questions: where state lives (four separate places, only one of which is the database), and what survives an update (the shared runtime paths, not the release directory).

Stack

The request path

There is no separate API server and no service mesh. A request from the browser hits a Next.js route handler, which does its own authorization and talks to Prisma directly:

browser
  → Next.js route handler  (src/app/api/**/route.ts)
      ├─ getServerSession()        auth + role check, per route
      ├─ Prisma client             src/lib/db.ts
      ├─ domain library            src/lib/**  (sequencing, pipelines, ena, files, …)
      └─ filesystem                only under the configured data base path
  → JSON response

Two consequences worth internalising:

• There is no middleware. Every route repeats its own session and role check. That is why the same failure mode can be a 401 on one endpoint and a 403 on another, and why adding a route means adding its guard.
• Pipelines are out of process. SeqDesk writes a run.sh and launches it, either as a detached local process or via sbatch. Nothing about a running workflow lives in the web process, which is why the app can be restarted mid-run without killing the run.

Where state lives

Four independent stores. Losing track of which is which is the most common cause of “the setting I changed did nothing”.

Pipeline reference databases (GTDB, Kraken2 indexes, the MetaxPath bundle) are a fifth store in practice, pointed at by pipelines.databaseDirectory and by per-pipeline config keys. They are large, shared, and deliberately not inside the release directory.

On-disk layout of an installation

A packaged install is a release layout: several unpacked releases, a current symlink selecting the active one, and shared runtime paths that live above the releases and are symlinked into each one.

<install-dir>/
├── start.sh                    # cd current && exec ./start.sh
├── current -> releases/1.1.125
├── releases/
│   ├── 1.1.124/
│   └── 1.1.125/
│       ├── server.js           # Next.js standalone entrypoint
│       ├── start.sh            # resolves PORT + DATABASE_URL, then exec node server.js
│       ├── .next/              # build output and static assets
│       ├── node_modules/       # traced runtime dependencies only
│       ├── public/
│       ├── prisma/schema.prisma
│       ├── scripts/            # bundled workers + install/maintenance scripts
│       ├── settings.json    -> ../../settings.json
│       ├── data          -> ../../data
│       ├── pipelines     -> ../../pipelines
│       └── pipeline_runs -> ../../pipeline_runs
├── settings.json               # THE configuration file
├── data/                       # shipped reference data (sequencing tech, MIxS, …)
├── pipelines/                  # installed pipeline packages
├── pipeline_runs/              # run directories
├── .update-temp/               # staging area used by the in-app updater
├── package.json      -> current/package.json
└── package-lock.json -> current/package-lock.json

The shared set is exactly settings.json (or the legacy seqdesk.config.json, if that is what the install already had), data/, pipelines/, and pipeline_runs/. Both the installer and the in-app updater create the same links, and the application writes its configuration through the symlink, so current/settings.json and <install-dir>/settings.json are always the same file. If they ever diverge, the live configuration has split in two.

An update stages the new release under .update-temp/, re-creates the shared links inside it, moves it into releases/<version>/, and then flips current atomically by creating a temporary symlink and renaming it over the old one. Rolling back is the same flip in reverse — with one exception, spelled out in Manual Update & Rollback: database migrations are forward-only, so reverting the code symlink does not revert the schema.

A development checkout has no release layer. It is just the repository, with settings.json at the root:

seqdesk/
├── src/
│   ├── app/          # pages and API route handlers
│   ├── lib/          # domain libraries
│   │   ├── config/   # multi-source configuration
│   │   ├── pipelines/# packages, launcher, monitor, output resolution
│   │   ├── sequencing/# orders, uploads, run plans, delivery
│   │   ├── ena/      # XML generation and Webin submission
│   │   └── files/    # discovery, path safety, matching
│   └── types/
├── prisma/schema.prisma
├── pipelines/        # pipeline packages
├── data/             # shipped reference data
├── scripts/          # workers, build-release.sh, install helpers
└── settings.json

How a release is built

scripts/build-release.sh produces seqdesk-<version>.tar.gz. It is worth knowing what it does, because it explains what is and is not present on a production host:

1. Runs npm run build, which generates the Prisma client and builds the Next.js standalone output.
2. Copies .next/standalone, then prunes anything Next’s file tracing pulled in that does not belong in a release: src/, docs/, logs/, pipeline_runs/, coverage/, .nextflow/, Playwright output, .next/cache, any local *.db files. That is why you cannot read application source on a packaged install.
3. Re-adds the two runtime dependencies file tracing cannot infer: @prisma/client and bcryptjs.
4. Bundles the worker scripts (stream-monitor, pipeline-monitor, pipeline-cli) into standalone CommonJS with esbuild, so a release ships .js workers rather than TypeScript.
5. Copies public/, prisma/, pipelines/, seqdesk.config.example.json, package-lock.json, next.config.ts, and the install/maintenance scripts.
6. Removes private pipeline packages listed in SEQDESK_PRIVATE_PIPELINES (default: metaxpath) — a public tarball never contains them.
7. Writes start.sh and tars the whole directory, reporting size and SHA-256.

start.sh is where the runtime contract is enforced. It resolves the port from $1, $PORT, app.port in the config file, or the port in runtime.nextAuthUrl, defaulting to 3000; resolves DATABASE_URL and DIRECT_URL from the environment first and then from the config file’s runtime block; and refuses to start on a file: URL or anything that is not a postgresql:// / postgres:// connection string.

Configuration resolution

Configuration comes from four sources, highest priority first:

1. Environment variables (SEQDESK_*)   ← highest priority
2. Config file (settings.json)
3. Database (SiteSettings)
4. Built-in defaults                   ← lowest priority

This resolution happens in two stages, and the distinction matters when you are debugging:

• loadConfig() merges defaults ← file ← environment. It never touches the database, so it is safe to call before the database is reachable. The file is found by looking in the working directory for settings.json, then seqdesk.config.json, then .seqdeskrc, then .seqdeskrc.json — the first that exists wins.
• mergeWithDatabase() folds the SiteSettings row underneath that result, so a value present in the file or environment still wins over the one an admin edited in the interface.

Every resolved value carries a source tag — env, file, database, or default — which is what the admin screens display next to a setting, and what lets a value be shown as read-only when a file or environment variable has taken it over. The resolved configuration is cached for 60 seconds; the cache is also cleared explicitly after a settings write.

Some code paths deliberately refuse to accept a database value. The data base path, for instance, only honours site.dataBasePath when its source is env or file; otherwise it falls back to SiteSettings.dataBasePath, and only then to a local development directory.

Full details, including the complete environment variable list, are in Configuration.

The pipeline package contract

A pipeline is a directory under pipelines/<id>/ containing four declarative files. SeqDesk never hardcodes a workflow; it reads the package and derives everything else.

manifest.json

The manifest is what makes a package safe to run. Its outputs block declares not just where a file will be, but what SeqDesk is allowed to do with it:

{
  "manifestVersion": 1,
  "package": { "id": "fastqc", "name": "FastQC", "version": "0.1.0" },
  "files": { "definition": "definition.json", "registry": "registry.json",
             "samplesheet": "samplesheet.yaml",
             "scripts": { "discoverOutputs": "scripts/discover-outputs.mjs" } },
  "targets": { "supported": ["order"] },
  "inputs": [ { "id": "reads", "scope": "sample", "source": "sample.reads", "required": true } ],
  "execution": { "type": "nextflow", "pipeline": "./workflow", "profiles": ["conda"] },
  "outputs": [
    {
      "id": "sample_fastqc_reads",
      "scope": "sample",
      "destination": "sample_reads",
      "fromStep": "fastqc",
      "writeback": {
        "target": "Read",
        "mode": "merge",
        "fields": { "fastqcReport1": "fastqcReport1", "readCount1": "readCount1" }
      },
      "discovery": { "pattern": "fastqc_reports/*_R1_fastqc.html", "matchSampleBy": "filename" }
    }
  ],
  "schema_requirements": { "tables": ["Read", "PipelineRun", "PipelineArtifact"] }
}

writeback is an allowlist, not a suggestion. A package may only merge the fields it names, onto the model it names — which is why a QC pipeline can update fastqcReport1 and avgQuality1 on a Read but cannot touch file1. Packages that produce replacement read files declare a candidate output instead, and those land as staged PipelineArtifacts awaiting admin promotion rather than as canonical reads. See pending writebacks.

samplesheet.yaml

The generated input is built from the database, never from whatever happens to be on disk:

samplesheet:
  format: csv
  filename: samplesheet.csv
  rows:
    scope: sample
  columns:
    - name: sample_id
      source: sample.sampleId
      required: true
    - name: fastq_1
      source: read.file1
      required: true
      transform:
        type: prepend_path
        base: '${DATA_BASE_PATH}'

source addresses canonical SeqDesk records (sample.*, read.*, study.*), and prepend_path is what turns a stored relative path into an absolute one for Nextflow. Because the samplesheet is derived, a run is always consistent with the records at launch time — and a run’s config is snapshotted onto the PipelineRun row so later changes to the defaults do not rewrite history.

Pipeline execution flow

Study or Order context
  → Package resolution (manifest + registry + definition)
  → Validation (metadata → inputs → config → derived config)
  → Run record created (status: pending)
  → Prepare: run number, run directory, samplesheet.csv, nextflow.config, run.sh
  → Execute: detached `bash run.sh`, or `sbatch --parsable run.sh`
  → Monitor: weblog events + trace file + scheduler queue state
  → Output discovery: artifact scan + optional discoverOutputs script
  → Resolution: Assembly / Bin / PipelineArtifact rows, allowlisted Read writeback

Three sources report on a running pipeline, and they are deliberately redundant:

PipelineRun.statusSource records which one last wrote the status, which is the first thing to check when a run’s state looks wrong. On an isolated cluster the weblog never arrives, lastWeblogAt stays null, and finalization has to come from the trace plus the exit-code marker run.sh writes.

Background workers

Helper processes run outside the web server. They are declared in a static registry (src/lib/workers/registry.ts), spawned from API routes, and tracked as BackgroundWorkerProcess rows carrying PID, log path, and exit code. The Admin → Background Workers page renders one card per registry entry.

Two further entries — stream-simulator and discover-simulator — are marked devOnly and are hidden in production builds.

Both real workers are shipped as bundled CommonJS in scripts/ and coordinate through database fields (StreamRun.monitorId, heartbeatAt; workerPause in SiteSettings.extraSettings) rather than any in-process mechanism. The web server can therefore restart without orphaning them, and a dead monitor is detectable by a stale heartbeat.

ENA submission flow

Study + Samples (validated)
  → XML generation (Study, Sample, Submission XMLs)
  → HTTP POST to the ENA Webin API (Basic Auth, test or production endpoint)
  → Receipt XML parsing
  → Accession writeback (Study.studyAccessionId, Sample.sampleAccessionNumber)
  → Submission row: status + full receipt + step timeline

Test and production are separate worlds: a test submission stamps Study.testRegisteredAt and never sets submitted. Concurrency is guarded by a PostgreSQL advisory lock plus an existence check, so the same study cannot be submitted twice at once. See ENA Submission.

Security model

Related reading

• Data Model — what the database actually holds
• App API — the endpoints this architecture exposes
• Configuration — every setting and where to put it
• Installation — getting the layout above onto a host
• Updates & Maintenance — how current gets flipped