Astronomer
Managed Apache Airflow service for orchestrating and monitoring data pipelines in the cloud
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Fast, flexible Go library for building robust scientific pipelines
SciPipe lets you compose command‑line tools into efficient, parallel, and reproducible workflows using Go’s type‑safe code, with fine‑grained file naming, streaming, and built‑in audit reporting.
SciPipe is a Go library that enables developers to create scientific workflows by connecting command‑line programs and native Go processes through a flow‑based programming model. Workflows are defined in Go code, compiled into fast binaries, and support parallel execution, task‑level concurrency, and streaming to minimize disk usage. Built‑in audit logs record every command run, making pipelines reproducible and allowing easy restarts without overwriting existing results.
Install SciPipe via the Go toolchain (go install github.com/scipipe/scipipe/...@latest) and initialize a module for your workflow. Once written, the workflow can be run directly with go run or compiled into a self‑contained executable for distribution across platforms where Go is supported. The library works equally well for bioinformatics, cheminformatics, and any domain that relies on chained command‑line tools.
When teams consider SciPipe, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Genome variant calling pipeline
Orchestrates alignment, sorting, and annotation tools with parallel execution, producing reproducible results and audit logs.
Chemical property prediction
Streams molecular data through custom Go analysis and external prediction tools, reducing intermediate storage.
Large‑scale image processing
Chains image conversion utilities while streaming data, achieving high throughput without excessive disk I/O.
Reproducible data‑science experiment
Generates detailed audit reports and enables easy restart of interrupted runs, ensuring full reproducibility.
Install Go, then run `go install github.com/scipipe/scipipe/...@latest` and create a Go module for your workflow.
Workflows are defined in Go; however, the `scipipe new` command can scaffold a basic Go file to get started quickly.
No, CWL support is not currently implemented.
It creates an audit JSON file for each output and can restart interrupted runs without overwriting existing results.
Any platform supported by Go; compiled binaries are portable across operating systems and architectures.
Project at a glance
DormantLast synced 4 days ago
