Astronomer
Managed Apache Airflow service for orchestrating and monitoring data pipelines in the cloud
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Visual, code‑first data pipelines without YAML or frameworks
Build, version, and run Python, R, or Julia pipelines via a UI, notebooks, and Git, with flexible environments and service orchestration.
Orchest lets data scientists and ML engineers design end‑to‑end pipelines without writing YAML or learning a new DSL. Using a drag‑and‑drop UI, you can connect notebook cells or script files written in Python, R, or Julia, define dependencies, and run the workflow on demand or on a schedule.
Pipelines run in isolated environments that can be backed by local Docker, Kubernetes, or the managed Orchest Cloud service. Long‑running services (e.g., databases or web servers) can be launched once per pipeline execution and accessed by subsequent steps. Projects are versioned with Git, enabling reproducible experiments. Although Orchest is in beta and no longer actively developed, it remains usable for prototyping and small‑scale production, with Apache Airflow suggested for larger, long‑term needs.
When teams consider Orchest, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Train and compare regression models
Iterate on multiple models within a single visual pipeline, tracking performance and code versions.
Run dbt transformations alongside Python
Combine SQL‑based dbt models with custom Python steps for hybrid data processing.
Execute PySpark jobs
Process large datasets using Spark within the orchestrated workflow, leveraging defined environments.
Connect to external databases via SQLAlchemy
Pull, transform, and load data from relational sources in a reproducible, version‑controlled pipeline.
Development has stopped; the project is in beta and receives no new features, though it remains usable.
You can self‑host using Docker or Kubernetes, or use the managed Orchest Cloud service.
Python, R, and Julia can be used directly in notebooks or scripts.
Projects are linked to a Git repository, allowing you to commit pipeline code and track changes.
Yes, Orchest Cloud provides a managed environment for running pipelines without self‑hosting.
Project at a glance
DormantLast synced 4 days ago
