Astronomer
Managed Apache Airflow service for orchestrating and monitoring data pipelines in the cloud
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Programmatically author, schedule, and monitor workflows as code
Apache Airflow is a platform for orchestrating complex workflows through code-defined DAGs, featuring a rich UI, extensible operators, and robust scheduling for batch data pipelines.
Apache Airflow is a platform designed for teams that need to programmatically author, schedule, and monitor complex workflows. By defining workflows as code (DAGs), Airflow makes pipelines maintainable, versionable, testable, and collaborative—eliminating the brittleness of GUI-based workflow tools.
Airflow excels at orchestrating mostly static, slowly changing workflows where tasks are idempotent and delegate heavy computation to external systems. The scheduler executes tasks across worker arrays while respecting dependencies, and the rich web UI provides real-time visibility into pipeline execution, progress monitoring, and troubleshooting. While not a streaming platform, Airflow is commonly used to process real-time data by pulling from streams in batches.
With dynamic pipeline generation through Python code, Jinja templating for customization, and a wide library of built-in operators, Airflow adapts to diverse orchestration needs. Tested against PostgreSQL, MySQL, Kubernetes, and multiple Python versions, it supports both AMD64 and ARM64 platforms. The project is maintained by the Apache Software Foundation and widely adopted across industries for data engineering, ETL, and ML pipeline orchestration.
When teams consider Apache Airflow, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
ETL Pipeline Orchestration
Coordinate extraction, transformation, and loading across databases, data warehouses, and cloud storage with dependency-aware scheduling and retry logic.
Machine Learning Workflow Automation
Orchestrate model training, validation, and deployment pipelines with parameterized DAGs that integrate with MLOps tools and compute clusters.
Batch Processing from Streaming Sources
Pull data from Kafka or other streams in scheduled batches, process through idempotent tasks, and load into analytics platforms.
Multi-System Data Integration
Coordinate data movement and transformations across APIs, databases, and SaaS platforms using extensible operators and custom hooks.
No, Airflow is not a streaming solution. However, it is commonly used to process real-time data by pulling from streams in batches on a schedule.
Airflow supports PostgreSQL (13-17), MySQL (8.0, 8.4, Innovation), and SQLite (3.15.0+). SQLite is only for development; PostgreSQL or MySQL are recommended for production.
For production, only POSIX-compliant systems (Linux) are supported. On Windows, use WSL2 or Linux containers for development and testing.
Airflow keeps dependencies open for flexibility, which can cause conflicts. Use the provided constraint files from the constraints-main or constraints branches for repeatable installations.
Tasks should not pass large data volumes directly. Use XCom for metadata only, and delegate data-intensive operations to external systems like data warehouses or processing frameworks.
Project at a glance
ActiveLast synced 4 days ago
