Astronomer
Managed Apache Airflow service for orchestrating and monitoring data pipelines in the cloud
Conductor
Scalable orchestration engine for resilient microservice workflows
Conductor lets developers define event‑driven workflows as JSON, offering automatic retries, flexible task types, built‑in UI, and support for Redis, MySQL, Postgres, Elasticsearch, and more.
Overview
Who should use Conductor?
Conductor targets development teams building distributed microservice or event‑driven applications that need reliable, durable workflow execution. It is suited for enterprises and startups alike that want to decouple services while maintaining end‑to‑end visibility.
Core capabilities
Workflows are expressed as versioned JSON, enabling a code‑first approach. The engine ships with a rich catalog of task types—HTTP, Lambda, Sub‑workflow, Event, and more—plus a customizable web UI for monitoring, debugging, and observability. Automatic retries, fallback mechanisms, and configurable timeouts provide resilience. Persistence and queue layers are pluggable, supporting Redis, MySQL, Postgres, and Elasticsearch (or Opensearch) out of the box. Official SDKs for Java, Python, JavaScript/TypeScript, Go, and C# simplify client integration.
Deployment options
Developers can start locally with Docker Compose, requiring Docker Desktop, Java 17+, and Node 14 for UI builds. For production, Conductor can be built from source and run as a standalone Java service on any JVM‑compatible host, with flexible database and indexing configurations. The project is actively maintained by Orkes and a vibrant open‑source community.
Highlights
Workflow as code with JSON versioning
Rich task catalog (HTTP, Lambda, Sub‑workflow, Event)
Built‑in UI for monitoring and debugging
Pluggable persistence and queue backends (Redis, MySQL, Postgres, Elasticsearch)
Pros
- Native support for retries and fallback handling
- Horizontal scalability for high‑traffic workloads
- Official SDKs for major programming languages
- Comprehensive observability via UI and metrics
Considerations
- Requires Java 17+ runtime
- Operational overhead for self‑hosting databases and Elasticsearch
- Learning curve for JSON‑based workflow definitions
- May be overkill for very simple automation needs
Managed products teams compare with
When teams consider Conductor, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Fit guide
Great for
- Teams building distributed microservice architectures
- Enterprises needing durable, fault‑tolerant workflow execution
- Developers who prefer code‑first workflow definitions
- Organizations able to manage their own infrastructure
Not ideal when
- Simple scripts that could use basic cron jobs
- Teams without capacity to maintain Java services
- Projects requiring a fully managed SaaS orchestration solution
- Environments lacking required databases or Elasticsearch
How teams use it
Order fulfillment pipeline
Coordinates inventory check, payment processing, shipping label creation, and notification across services with automatic retries on failures.
Data ingestion and transformation
Orchestrates fetching data from APIs, applying Lambda tasks for transformation, storing results in a data lake, and handling back‑pressure via queue integration.
CI/CD multi‑stage deployment
Runs build, test, security scan, and deployment tasks across heterogeneous environments, rolling back automatically on errors.
Customer onboarding workflow
Guides new users through verification, account provisioning, welcome email, and CRM update, with real‑time monitoring via the UI.
Tech snapshot
Java77%
Groovy15%
JavaScript8%
TypeScript1%
SCSS1%
PLpgSQL1%
Frequently asked questions
What languages can I use to interact with Conductor?
Conductor provides official SDKs for Java, Python, JavaScript/TypeScript, Go, and C#.
Do I need to run Elasticsearch?
Elasticsearch is the default indexing backend, but you can use Opensearch or disable indexing if not required.
Can I run Conductor without Docker?
Yes, you can build the Java application from source and deploy it on any JVM‑compatible host.
How does Conductor handle task failures?
It supports configurable retries, timeout policies, and fallback tasks to ensure resilient execution.
Is there a UI for monitoring workflows?
A built‑in web UI lets you view workflow instances, task status, and logs, and it can be customized.
Project at a glance
Active- Stars
- 31,571
- Watchers
- 31,571
- Forks
- 807
LicenseApache-2.0
Repo age2 years old
Last commit2 days ago
Primary languageJava
Last synced 3 hours ago