Facility-managed Kraken2 database for read cleaning
- Read-cleaning now exposes a Kraken2 contaminant database in the pipeline database manager, mapped to --kraken2db; link an existing directory on a shared mount or pull a published index.
- When a facility has configured the Kraken2 database, the per-order sequencing form hides the 'Kraken2 DB' field while still applying the managed path to the run.
- Added read-cleaning runtime defaults (Kraken2 classification and contaminant taxon) to the install-profile editor.
Reliable pipeline status and run control
- The pipeline status monitor auto-starts on boot, so finished SLURM/local jobs are reconciled and no longer stay stuck showing 'running' after an app restart or update.
- Added a 'Stop run' action to the order and study pipeline menus to cancel an active run (SLURM scancel or local process).
Pipeline run reliability and output integrity
- Finished pipeline jobs are reconciled against the live scheduler, so they no longer hang showing 'Running 99%'.
- A failed step is never masked by a sibling task's success, and a successfully retried step recovers instead of staying failed.
- Concurrent output resolution can no longer create duplicate artifacts, assemblies, or bins.
- Read-cleaning Kraken2/BBDuk database paths are validated before a run launches.
- Hardened SLURM job submission, run-number generation past 999, and samplesheet field escaping.
Publish pipeline and sequencing delivery results
- Admins can mark order pipeline runs visible to users and hide them again.
- Admins can publish cleaned sequencing reads and customer-facing reports for order owners.
- Owner file preview/download routes now enforce published-delivery access gates.
- Order overview shows released analysis results and sequencing files for the owner.
Pipeline final results and output previews
- Added explicit final-result selection for completed study/order pipeline runs.
- Added ranked run output links with compact file menus and report prioritization.
- Streamed pipeline output previews with size limits for HTML, PDF, text, and table files.
- Improved package-derived pipeline settings and runtime warning visibility.
Hosted profile reload and MetaxPath PlusPF fixes
- Fix hosted profile reload failing with a duplicate apply-install-profile helper declaration.
- Use MetaxPath 0.1.5 as the PlusPF runtime hardening baseline in pipeline readiness warnings.
- Pass Nextflow params files before explicit mapped params so UI/runtime overrides win over params-file defaults.
MetaxPath Runtime Warnings
- Warn when MetaxPath PlusPF uses Kraken2 without memory mapping.
- Warn when PRED_VFS_AMRS memory is below 96 GB or the installed MetaxPath package predates 0.1.4.
- Show non-blocking runtime warnings in admin readiness, study launches, and order launches.
Status indicator improvements
- Add Gemma demo dataset applied, changed, and missing status colors.
- Fix running pipeline status colors.
- Use accurate sidebar read and pipeline status dots.
Pipeline controls and sidebar status fixes
- Add execution target availability controls for pipeline runs.
- Harden footer pipeline load status handling.
- Preserve study page scroll position across updates.
- Show the ENA publishing sidebar dot as green only after registration.
Updater Prisma recovery
- Pin Prisma runtime tooling to 5.22.0 and block npx Prisma fallbacks.
- Add Retry update and Clear failed status controls for failed admin updates.
- Use locked runtime dependency installs during updater repair and release installs.
Pipeline profile and study scope fixes
- Keep MetaxPath on the study pipeline page while preserving package-declared targets.
- Apply hosted install-profile runtime defaults to saved pipeline configs used by UI, CLI, and API starts.
- Refresh runtime dependencies and Prisma during admin updater installs.
MetaxPath order compatibility
- Scope MetaxPath compatibility checks to the actual run target, so order-scoped runs no longer require study support.
- Keep stale package version and removed --chunk-breadth compatibility guards in place.
- Publish SeqDesk v1.1.104 for updater-based rollout after syncing MetaxPath package v0.1.2.
Local Pipeline CLI
- Added local seqdesk pipeline commands for listing, starting, watching, syncing, logging, debugging, output inspection, and cancellation.
- Shared pipeline run services now keep API and CLI behavior aligned for create/start/status/sync/logs/debug/output/cancel flows.
- Added MetaxPath compatibility checks to block stale packages with removed CLI flags or missing study target support.
- Grouped configuration source settings and help text in the admin settings view.
MetaxPath CLI compatibility
- Publish SeqDesk v1.1.102 release metadata and installer package.
- MetaxPath package v0.1.1 removes the unsupported --chunk-breadth flag from METAX_PROFILE while keeping min-breadth behavior.
- After syncing the MetaxPath GitHub package, order-scoped smokes should proceed past the Metax CLI invocation.
MetaxPath order-run compatibility
- Normalize installed MetaxPath descriptors so order-scoped runs are available from order analysis.
- Map MetaxPath paramsFile to Nextflow -params-file instead of --paramsFile.
- Set conda.useMamba = false in generated Nextflow config so conda is used when mamba is unavailable.
Hosted profile reload and pipeline setup visibility
- Add an admin action to reload the recorded hosted install profile, optionally applying profile assets.
- Show resolved database/profile-managed pipeline settings in the pipeline configuration dialog, including MetaxPath params file paths.
- Redirect the legacy order pipeline page and note mention links to the sequencing analysis view.
Pipeline runtime smoke test
- Add npm run pipeline:e2e:runtime for command-line local and SLURM pipeline validation.
- Use the admin dummy-data order automatically, with --ensure-dummy-data to seed it when missing.
- Document per-pipeline SLURM runtime policy and headless smoke-test usage.
Pipeline SLURM runtime policy
- Add global, per-pipeline, and per-run execution policy resolution for local vs SLURM runs.
- Persist resolved execution mode/profile on pipeline runs and keep local execution independent of SLURM.
- Add admin runtime controls and run-dialog execution target selection.
- Accept per-pipeline execution overrides from install profiles and infrastructure imports.
- Add a real dev-cluster SLURM smoke command with focused API/runtime test coverage.
Clean release packaging
- Strip local work directories, pipeline run outputs, test reports, coverage, local databases, and pipeline download state from release tarballs.
- Remove Next.js trace manifests from public release artifacts.
- Keep seqdesk --version as a launcher-level version command instead of forwarding it to the installer.
Admin asset visibility and CLI asset apply
- Add seqdesk assets apply for applying hosted profile pipeline database assets and seed fixtures to an existing install.
- Add a FACILITY_ADMIN activity API and persistent footer activity status for DB downloads and example dataset loads.
- Surface progress, speed, ETA, sanitized log excerpts, and clearer failure messages for MetaxPath DB downloads.
- Track Gemma MetaxPath example dataset loading as a background activity and show storage-path errors in admin settings.
Installer URL clarity
- Separate browser URL, local health URL, and bind host in install output.
- Default generated start scripts to bind on 0.0.0.0 unless SEQDESK_BIND_HOST is set.
Updater buffer hotfix
- Run updater shell commands with an expanded output buffer to avoid Node exec maxBuffer failures.
- Make Prisma migrate deploy quiet by default so older updater versions can install this hotfix without overflowing stderr.
Gemma MetaxPath demo dataset
- Add an in-app Admin Settings action to download and seed the Gemma Nanopore MetaxPath dataset.
- Extend install-profile fixtures to seed metadata-driven example datasets with cleaned FASTQ read metadata.
- Add a development install-profile toggle for the hosted Gemma dataset.
Use existing file for pipeline reference databases
- Add Use existing file button to the Download DB dialog that writes the path into pipeline config and marks the database as downloaded.
- Add /api/admin/settings/pipelines/download-db/link-existing endpoint that validates the absolute path, stats the file, and updates the download record.
Pipeline DB download dialog with pre-flight, cancel, and bandwidth limit
- Add Download DB confirmation dialog with editable target path and source URL preview.
- Pre-flight remote size and free disk space at the target; block download when space is insufficient.
- Detect partial downloads and let the admin choose Resume or Start over.
- Cancel a running database download from the card or the progress block.
- Optional sha256 checksum verification after download with a Verifying/Installing phase indicator.
- Optional per-download bandwidth limit (e.g. 10M, 512K) passed to curl/wget.
Bundle worker scripts so tsx isn't required at runtime
- Bundle scripts/stream-monitor.ts and scripts/pipeline-monitor.ts via esbuild in build-release.sh.
- Worker spawner prefers bundled .js next to the .ts source, falling back to tsx for dev.
Abort update if migrations cause data loss
- Snapshot order/sample/study/user counts before and after migrations during in-app update.
- Abort and restore from .update-backup if any of those counts dropped.
Expanded MinKNOW stream ingest and pipeline DB download UI polish
- Add StreamIngestedFile model with sample-link migrations and per-stream-run barcode lookup API.
- Expand MinKNOW stream monitor with ingest tracking, diagnose scripts, and admin diagnose API.
- Add fastq-stats library and tests for read/base counting from ingested files.
- Add seed library and admin dummy-data API for development setup.
- Surface progress bar, speed, ETA, target path, and source URL on pipeline DB downloads.
- Polish admin settings, background workers, notifications, and sequencing UI.
Private GitHub release assets in MetaxPath installer
- Add Accept: application/octet-stream to the curl and wget paths in scripts/install-private-metaxpath.sh so private GitHub release-asset URLs return the binary tarball instead of JSON metadata.
- Bump npm seqdesk launcher and root package version to 1.1.86 so npx seqdesk@latest pins this release.
Bundle pipeline setup script in release tarball
- Bundle scripts/setup-conda-env.sh in release tarballs so installs with pipelines enabled can complete the Conda environment step.
- Set placeholder DATABASE_URL in the GitHub Actions release workflow so Prisma generate + next build succeed end to end.
- Swap inverted pipelines.enabled assertions in the Alma install canary so the without-profile job asserts disabled pipelines and the with-profile job asserts enabled.
- Update tests for the new DatabaseStatus.reason field and the full-runNumber rendering in the order pipeline view.
Hosted install profiles and managed bootstrap accounts
- Apply install.dir and install.usePm2 from hosted profile configs so managed installs need fewer command-line flags.
- Support profile-provided bootstrap admin and researcher accounts while persisting only bcrypt password hashes.
- Use hosted profile runtime database URLs during install and seed fallback paths.
- Harden installer output by redacting database URL passwords and cleaning temporary resolved profile files.
Order-scoped pipeline targeting and read writeback
- Add manifest targets and Read writeback contracts for packaged pipelines, with validation and compatibility scope derivation.
- Update FastQC, FASTQ checksum, and Simulate Reads to write read metadata and generated read files through manifest mappings.
- Filter admin, store, sidebar, and order pipeline views by catalog so order pages only show order-scoped pipelines.
- Expand automated coverage across pipeline settings/routes, file APIs, sidebar endpoints, installer updates, and seed/runtime config loading.
JSON config runtime cleanup and docs pruning
- Migrate installer/update/release scripts to use runtime keys in seqdesk.config.json.
- Remove .env.example and clean README/install references to environment-file setup.
- Prune public docs and internal artifacts from this repository.
- Include test and pipeline helper cleanups from recent commits.
Configurable sequencing-tech sync and test hardening
- Admin sequencing tech settings now include a configurable registry sync URL used for reset and update checks.
- Sequencing-tech API now validates and persists sync URLs, tracks last sync time, and returns clearer update status messages.
- Added ENA API client tests for accession extraction and authentication error mapping.
- Expanded tests for pipeline manifest/package loading, generic adapter/executor paths, and parser/runtime validation.
UX refresh for studies, files, and admin forms
- Redesigned the Manage Files workflow with tabbed navigation and improved order file handling.
- Refined study and order detail pages, including updated overview/status presentation and editing flows.
- Polished dashboard surfaces (analysis, help, messages, settings) and sidebar navigation.
- Updated admin form builder, module management, sequencing technology, and study-form builder interfaces.
Installer hardening and private pipeline setup
- Add admin workflow to install private pipeline packages from Settings > Pipelines.
- Support hosted private MetaxPath package metadata while excluding private pipelines from public release tarballs.
- Installer no longer auto-runs sudo system package installs and now prints manual dependency commands.
- Ensure distribution installs have runtime Node modules, preventing next command-not-found startup failures.
- Reconfigure now skips DB push/seed by default and supports explicit reseed via --reseed-db.
- Respect configured condaPath for Miniconda install target and improve conda init shell guidance.
Study notes reliability and API validation
- Prevent unsaved notes edits from being overwritten when study data refreshes.
- Validate study update request body types and return clear 400 errors for invalid payloads.
- Improve title/description/alias normalization to avoid unsafe trim operations on non-string inputs.
Installer automation and workflow updates
- Added non-interactive installer support with --config and -y, plus reconfigure mode for in-place updates.
- Installer now syncs runtime settings and optional secrets from config/.env with safer automation defaults.
- Improved sequencing workflow UX with barcode-aware Excel import helpers and validation tooling.
- Refined dashboard and admin pages for orders, studies, and settings workflows.
SubMG coverage from metadata
- Read assembly coverage from sample/order custom fields (coverage_depth and related keys) for SubMG YAML generation.
- Use pipeline config assemblyCoverageValue/coverageValue when metadata is absent, and only then fall back to 1.
SubMG assembly coverage field fix
- Add required ASSEMBLY COVERAGE_VALUE to SubMG config YAML (default 1, configurable).
- Keep SubMG read submission flow intact while unblocking assembly step validation.
SubMG ENA compatibility fixes
- Normalize ONT platform values (including MinION/GridION/PromethION) to ENA-valid OXFORD_NANOPORE in SubMG manifests.
- Add FASTQ file path fallback between .fastq/.fastq.gz and .fq/.fq.gz during SubMG run preparation.
Fix SubMG read manifest insert size
- Write a positive INSERT_SIZE in PAIRED_END_READS manifests (default 300, configurable via pipeline config insertSize).
- Stop defaulting to --skip_checks so SubMG preflight validation runs unless explicitly disabled.
Fix SubMG metadata false-missing when MIxS keys are prefixed
- Accept legacy _mixs_ prefixed checklist keys when validating SubMG required metadata.
- Write MIxS per-sample metadata using canonical checklist keys (without _mixs_ prefix) on study create/edit save.
Show exact missing SubMG metadata fields on pipeline card
- Include missing SubMG metadata field names in the card-level Missing summary.
- Include field-level details in the card check details line for Sample metadata.
SubMG metadata visibility and edit-study save warnings
- Show exact missing SubMG sample metadata fields in pipeline validation summaries and per-sample rows.
- Add warning dialog on Edit Study Review and Save for missing required sample fields, with Save Anyway option.
SubMG checksum and ENA target improvements
- Show ENA target server (test vs production) on SubMG cards and in the run dialog.
- Add Compute and add read checksums action for missing read MD5 values, including batch processing and status feedback.
- Prevent preferred assembly UI rollback when the save succeeded but metadata refresh fails.
- Keep checksum retry path available by applying local checksum overrides only after successful SubMG validation refresh.
Assemblies view and metadata validation
- Add Assemblies page/API and optional user assembly download access control
- Validate selected-sample metadata server-side before creating/starting pipeline runs
- Support sequencing-technology-derived platform resolution and MAG run-target restrictions
Study page not found fix
- Fix study detail lookup when URLs carry stale order-linked identifiers.
- Normalize study detail/edit/metadata pages to canonical study IDs.
Order detail load fix
- Refactors /api/orders/[id] to use explicit, resilient relation loading.
- Prevents order page hard failures when some linked records are missing or out of sync.
Simulation & analysis observability
- Add raw weblog events API endpoint for pipeline runs and link it in Analysis run details.
- Improve sequencing-file simulation routes and configurable template source handling.
- Add order file inspection API and strengthen file discovery/checksum workflows.
- Refresh admin settings and dashboard sections for clearer pipeline/file operations.
MAG stability + run UI
- Fix MAG CONCOCT env by pinning compatible Python and setuptools for CONCOCT steps
- Redesign analysis run page with Live Activity/Files/Details/Health tabs
- Populate Pipeline Files size data from run artifacts and filesystem
Pipeline reliability and SubMG clarity
- Improved large DB download retry/resume behavior and partial-transfer handling
- Prevented false DB-ready status when only partial files exist
- Added manual terminal fallback commands for DB downloads in admin pipeline settings
- Added per-pipeline missing-input summaries on study pipeline cards
- Added SubMG submission scope panel with include/exclude guidance and sample-level missing input details
Pipeline DB downloads + better MAG dummy reads
- Add Download DB action for MAG pipeline with progress/status in admin pipeline settings.
- Persist downloaded GTDB DB path into pipeline config automatically.
- Improve simulated FASTQ generation to reference-based paired-end reads for more reliable MAG assembly tests.
- Bump release to v1.1.41.
Pipeline runtime stability fixes
- Force Nextflow conda creation to use --override-channels with conda-forge and bioconda (no defaults).
- Generate unique Nextflow run names per run to avoid duplicate -name collisions.
- Record weblog event timestamps at receipt time to keep Event Feed relative times accurate.
- Keep SLURM launcher honoring configured queue and slurmOptions.
SLURM submission reliability
- Fix SLURM run.sh generation so #SBATCH directives are emitted before shell commands.
- Pass configured queue and slurmOptions (e.g. --qos=broker) directly to sbatch at submit time.
- Keep Nextflow SLURM process config enabled with executor, cpus, memory, queue, time, and clusterOptions from settings.
SLURM executor defaults
- Set process.executor, cpus, memory, queue, time, and clusterOptions from app SLURM settings.
- Accept clusterOptions alias in infrastructure import JSON.
- Update infrastructure setup JSON examples with SLURM fields and qos option.
Runtime and MAG Launch Fixes
- Prefer seqdesk-pipelines env during conda auto-detect to avoid defaulting to base.
- Use runtime platform detection from conda and avoid false macOS ARM runtime checks.
- Fix MAG flag generation to skip empty path values (prevents --gtdb_db resolving to 'true').
Updater DB preservation fix
- Fix updater SQLite path resolution to match Prisma (prisma/dev.db).
- Preserve DB backup/restore paths to avoid filename collisions.
- Improve restart fallback behavior for supervised deployments.
- Show auto-seed/setup errors directly in setup status UI.
Installer seeding reliability fix
- Installer no longer prompts for data/run paths during minimal setup.
- NEXTAUTH_URL is auto-derived from selected port.
- Seed scripts no longer require runtime bcryptjs module resolution.
- Release build now includes bcryptjs in packaged node_modules.
Auto-seed database on first launch
- Auto-seed database within the app on first launch when tables exist but data is missing.
- Install scripts gracefully handle seed failures with fallback chain.
- Setup page auto-detects seeding progress and redirects to login when ready.
Fix installer seeding with Prisma 7 environments
- Pin installer Prisma CLI usage for db push and db seed to project Prisma version.
- Ensure install-dist performs seeding reliably during installation.
- Update setup page instructions (remove legacy v2 path and use correct seed command).
Fix installer login/setup loop
- Fix install-dist: run prisma db seed during installation.
- Make /setup and /api/setup/status dynamic/no-cache to avoid stale redirect behavior.
- Use no-store setup status checks on login/home.
- Default pipeline support prompt to yes in install scripts.
SLURM Bootstrap Fixes & Debug Session Info
- Harden conda/nextflow bootstrap in generated SLURM and local run scripts.
- Add fallback to conda run for nextflow when PATH lookup fails.
- Add analysis Run Health panel with failure-first diagnostics.
- Add Copy session info action with plain-text debug bundle for support.
- Always load logs for failed and non-running runs in Live Log Viewer.
Admin UX Consistency & Workflow Updates
- Redesigned admin settings and infrastructure pages with consistent layout and clearer actions.
- Improved form-builder and study-form-builder usability with stronger save/test flows.
- Refined modules configuration to better reflect field enablement behavior.
- Enhanced ENA configuration UX, validation, and submission-history robustness.
- Aligned key dashboard/admin views for better visual consistency and mobile behavior.
Pipeline Execution Fixes & Analysis UX
- Fix SLURM scripts not redirecting output to monitored log files.
- Fix SLURM log file detection using wrong directory path.
- Fix incorrect status comparison in pipeline monitor.
- Fix run number generation NaN risk from malformed data.
- Fix output resolver masking errors as success.
- Fix always-true status note condition in order updates.
- Guard against undefined runId in pipeline start flow.
- Add try-catch for JSON.parse preventing page crash on malformed data.
- Remove duplicate sync intervals on analysis detail page.
- Show Analysis page to researchers (not just admins).
- Improve analysis empty state with guidance and link to studies.
Fix SLURM Log Directives
- Fix SLURM script log output/error path directives.
Pipeline Run Directory Validation
- Prevent root (/) as pipeline run directory.
- Add validation at save and start time.
- Clear error message for misconfigured settings.
SLURM and Null-Safety Fixes
- Allow pipeline start when conda not found locally (SLURM provides it).
- Use sbatch --parsable for reliable job ID parsing.
- Handle missing user/sample data gracefully on analysis pages.
Better SLURM Diagnostics
- Capture and display SLURM stderr output when sbatch fails.
- Verify sbatch command exists before attempting submission.
- Verify run script exists before submission.
Analysis Page Fix
- Fix crash when viewing analysis runs with incomplete data.
- Add null safety for sample reads and pipeline steps.
Safe Updates (Data Preserved)
- CRITICAL: Remove dev.db from release tarball to prevent data loss on update.
- Added backup/restore of database during update process.
- Fix client-side crash when viewing studies with samples that have no reads.
Update Safety + Study Page Fix
- Robust database backup and restore during self-update to prevent data loss.
- Fix client-side crash when viewing studies with samples that have no reads.
Flexible Study Creation
- Per-sample required fields now show warnings instead of blocking.
- Added 'Create Anyway' option when required fields are missing.
- Blur active inputs before validation to commit pending values.
Safer Updates
- Strip database files from update tarball to prevent data loss.
- Load DATABASE_URL from .env for custom SQLite paths.
- Skip migrations by default for SQLite unless SEQDESK_AUTO_MIGRATE=1.
File Management + Report Preview
- Add bulk delete for files page with confirmation dialog.
- Add HTML report preview for pipeline output files.
- Improve simulated reads: realistic sequences, Illumina headers, varied quality scores.
- Fix BigInt serialization crash on analysis detail page.
- Fix stale 'Last event' timestamps for queued and pending runs.
- Add Read Files tab to study detail page.
- Refactor pipeline section with card layout and history table.
- Order Progress step 2 reflects actual file state.
Queue Tracking + Event Feed
- Persist queue status + reason for SLURM and local runs.
- Record weblog events and show an event feed in Analysis.
- Track last weblog/trace timestamps in run details.
- Detect SLURM log files and surface them in Pipeline Files.
Pipeline Run Tracking
- Improved pipeline run status tracking and log previews.
SLURM Test Button
- Added a SLURM test button in Data & Compute.
Pipeline Run Visibility
- Record last pipeline events and status sources.
- Sync trace data to finalize running jobs.
- Live refresh + clearer running state in analysis views.
Pipeline code downloads
- Add background Nextflow pull with status tracking
- Show code cache status and download actions in pipeline settings
- Expose pipeline download state via admin API
Installer Env Fallback
- Ship .env.example in release tarball
- Installer creates minimal .env when missing
Update Progress & Restart Detection
- Persist update progress and show live status in Admin Settings
- Detect installed vs running version with restart pending state
- Show app version in the sidebar
Installer and Packaging Fixes
- Include Prisma client generator in release package
- Prompt for app port during install
- Start script respects PORT or .env
- Do not ship .env or dev.db in release
Installer Improvements
- Added preflight summary and configuration review
- Optional Node.js auto-install via nvm
- Cleaner error reporting and install summary
- Include Prisma client generator in release package
- Prompt for app port during install
- Start script respects PORT or .env
- Do not ship .env or dev.db in release
Install Flow
- Added interactive install wizard for curl-based installs
- Pipeline execution settings moved to Admin > Settings > Compute
- Simplified pipeline store UI with installed-first layout
Modern UI Design
- Modernized UI with warm neutral color palette
- Redesigned sidebar with improved navigation
- Inline field validation with connected error display
- Studies page redesigned to match Orders with search/filters
- Field Help panel with AI-like shimmer effect
- Full-width pages throughout the app
- Generic manifest-driven pipeline execution system
Pipeline Package System
- Added pipeline package system with store functionality
- Install pipelines directly from Admin Settings
- MAG pipeline available as first package
Update System and Admin Improvements
- Added disk space check before updates (requires 150MB free)
- Added update banner in dashboard for facility admins
- Added rollback documentation for manual recovery
- Interactive port selection when default port is in use
- Improved update progress indicators
Self-Update System
- Added self-update functionality in Admin Settings
- Version checking against seqdesk.com
- Automatic download and installation of updates
- Database backup before updates
Initial Release
- Order management with status tracking
- Sample metadata with MIxS compliance
- nf-core pipeline integration
- ENA submission support
- Multi-user authentication with roles