Aiven for Apache Flink
Fully managed Apache Flink service by Aiven.
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
High-performance resilient stream processor with declarative pipelines
Redpanda Connect lets you connect dozens of sources and sinks, apply Bloblang transformations, and guarantee at‑least‑once delivery—all via a single declarative YAML file, runnable as a binary or Docker image.
Redpanda Connect is a high‑performance stream processor that lets you stitch together a wide variety of data sources and sinks—AWS, Azure, GCP, Kafka, NATS, MQTT, SQL databases, and more—using a single declarative YAML configuration. The built‑in Bloblang mapping language enables complex enrichments, transformations, and filtering without writing code, while an in‑process transaction model guarantees at‑least‑once delivery even after crashes.
The processor can be run as a static binary (rpk connect run) or as a minimal Docker image, making it fit seamlessly into container‑orchestrated environments. Health endpoints (/ping, /ready) and extensive metrics (Prometheus, StatsD, JSON) simplify monitoring, and OpenTelemetry tracing provides visibility into each processing stage. Custom connectors are added via Go plugins, and extra components can be compiled with the x_benthos_extra build tag when needed.
It targets data‑engineering teams that need reliable, low‑latency pipelines without managing offset storage, and ops teams that value straightforward observability and cloud‑native deployment.
When teams consider Redpanda Connect, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Real‑time event enrichment from Pub/Sub to Redis Streams
Enriches incoming messages with computed fields using Bloblang and delivers them to Redis with at‑least‑once guarantees.
Batch file ingestion from S3 to ClickHouse
Pulls objects from S3, transforms records, and streams them into ClickHouse for analytics.
Cross‑cloud data sync between Azure Blob and GCP BigQuery
Continuously moves files from Azure Blob storage to BigQuery tables, handling format conversion on the fly.
Log aggregation from MQTT to Elasticsearch
Collects MQTT telemetry, filters noise, and indexes logs in Elasticsearch for search and visualization.
It uses an in‑process transaction model that acknowledges messages only after successful processing, without persisting state to disk.
Yes, you can download the static binary (rpk) for Linux or install via Homebrew and run it directly with a config file.
Metrics can be exposed to Prometheus, StatsD, or a JSON endpoint; health checks are provided via /ping and /ready, and OpenTelemetry tracing is emitted.
Write a Go plugin implementing the connector interface, import it via the public API, and build Redpanda Connect with the appropriate build tag.
Redpanda Connect guarantees at‑least‑once; exactly‑once requires external coordination and is not provided out of the box.
Project at a glance
ActiveLast synced 4 days ago
