Aiven for Apache Kafka
Managed Kafka with tiered storage and built-in schema registry.
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Scalable distributed event streaming platform for real‑time data pipelines
Apache Kafka delivers high‑throughput, fault‑tolerant event streaming, enabling real‑time analytics, data integration, and mission‑critical applications across heterogeneous environments for thousands of companies.
Apache Kafka is a distributed event streaming platform that lets developers build high‑performance data pipelines and real‑time analytics. It stores streams as immutable, partitioned logs, providing durability, replayability, and exactly‑once processing guarantees. Kafka’s core is written in Java and Scala, supporting Java 11‑17 runtimes and Scala 2.13, and offers client libraries for many languages.
Kafka can be run from the compiled binaries, via the provided Docker image, or integrated into CI/CD pipelines using Gradle tasks such as jar, test, and releaseTarGz. The project includes extensive testing support, coverage reporting, and auto‑generated documentation. For production, brokers are configured with kafka-storage.sh and started with kafka-server-start.sh. The ecosystem includes connectors, stream processing APIs, and tools for monitoring and scaling across clusters.
When teams consider Apache Kafka, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Log aggregation for microservices
Centralized, ordered event store enables traceability and fault‑tolerant communication between services.
Real‑time fraud detection
Stream processing on Kafka topics identifies suspicious patterns within seconds, allowing immediate response.
Change data capture (CDC) pipelines
Database change events are published to Kafka, feeding downstream analytics and search indexes.
IoT sensor data ingestion
High‑volume sensor streams are buffered in Kafka, providing reliable buffering before processing.
Kafka builds client modules with Java 11 release and other modules with Java 17; Java 11+ runtime is required.
Yes, the official Docker image (apache/kafka:latest) can be started with a simple port mapping.
Gradle tasks `javadocJar` and `scaladocJar` produce Javadoc and Scaladoc jars; `aggregatedJavadoc` creates a combined site.
Use Gradle’s `--tests` option with the module’s test task, e.g., `./gradlew clients:test --tests RequestResponseTest`.
Coverage HTML reports appear under each module’s `build/reports` directory, e.g., `core/build/reports/scoverageTest/index.html`.
Project at a glance
ActiveLast synced 4 days ago
