Astronomer
Managed Apache Airflow service for orchestrating and monitoring data pipelines in the cloud
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Event-Driven Declarative Orchestration Platform for Modern Workflows
Kestra orchestrates scheduled and event-driven workflows using declarative YAML. Build reliable data pipelines, microservices, and automation directly from the UI with Infrastructure as Code best practices.
Kestra is an event-driven orchestration platform designed for teams building reliable, scalable workflows. By combining Infrastructure as Code principles with an intuitive UI, Kestra enables engineers to define both scheduled and real-time workflows in simple YAML—no complex DAG programming required.
Define workflows declaratively using a built-in code editor with syntax highlighting, auto-completion, and validation. Execute tasks in any language (Python, Node.js, R, Go, Shell) locally, via Docker, or on Kubernetes. Trigger workflows on schedules or real-time events from Kafka, AWS SQS, Google Pub/Sub, file arrivals, and more. Integrate with cloud platforms (AWS, GCP, Azure), databases, APIs, and big data tools through hundreds of pre-built plugins.
Data engineers orchestrating ETL pipelines, DevOps teams automating infrastructure tasks, and platform engineers coordinating microservices rely on Kestra. The platform scales to millions of workflows with high availability, fault tolerance, and built-in Git version control. Every UI change automatically updates the YAML definition, ensuring workflows remain code-first while staying accessible to non-developers. Deploy via Docker, Kubernetes, or cloud-native services.
When teams consider Kestra, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Event-Driven Data Pipelines
Automatically trigger ETL workflows when new files arrive in S3 or messages land in Kafka, processing data in real-time without manual intervention.
Multi-Cloud Infrastructure Automation
Orchestrate provisioning and deployment tasks across AWS, GCP, and Azure using declarative workflows with built-in error handling and retries.
Microservice Coordination
Coordinate complex microservice interactions with conditional branching, parallel execution, and event triggers, maintaining resilience through timeouts and retries.
Scheduled Analytics Workflows
Run daily Python or SQL scripts against BigQuery, Snowflake, or Postgres, generating artifacts and sending Slack notifications on completion or failure.
Every change made in the UI automatically updates the underlying YAML definition. You can push workflows to Git directly from Kestra, ensuring version control and CI/CD integration regardless of how workflows are created.
Kestra supports Python, Node.js, R, Go, Shell, and other languages through its plugin ecosystem. Scripts execute locally, in Docker containers, or on Kubernetes depending on your task runner configuration.
Yes. Kestra supports real-time triggers for Kafka, AWS SQS, Google Pub/Sub, Azure Event Hubs, file arrivals, and custom events, enabling responsive automation alongside scheduled workflows.
Kestra is designed for high availability and fault tolerance, capable of handling millions of workflows. Deploy on Kubernetes or cloud platforms with horizontal scaling and distributed execution.
Docker is the recommended deployment method for local development and many production scenarios. Kestra also supports Kubernetes, Podman, and cloud-native deployments on AWS, GCP, and Azure.
Project at a glance
ActiveLast synced 4 days ago
