Aiven for Apache Flink
Fully managed Apache Flink service by Aiven.
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Fast, unified engine for large-scale data analytics
Apache Spark delivers a fast, unified analytics engine supporting Scala, Java, Python, and R, with built-in libraries for SQL, machine learning, graph processing, and streaming at scale.
Apache Spark is a unified analytics engine designed for large‑scale data processing. It offers high‑level APIs in Scala, Java, Python, and a deprecated R interface, enabling developers and data scientists to write applications in their language of choice. Built‑in libraries such as Spark SQL, MLlib, GraphX, and Structured Streaming extend the core engine to cover batch queries, machine‑learning pipelines, graph analytics, and real‑time stream processing.
Spark can run locally for development, on standalone clusters, or be managed by resource managers like YARN, Mesos, and Kubernetes. Integration with Hadoop storage systems allows seamless access to HDFS, Hive, and other compatible data sources. Users start interactive sessions via spark-shell (Scala) or pyspark (Python) and submit jobs with spark-submit or the example runner.
The project provides extensive documentation, a vibrant community, and a flexible build system based on Apache Maven, making it suitable for a wide range of big‑data workloads.
When teams consider Apache Spark, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Nightly data warehouse ETL
Processes terabytes of raw logs into curated tables within minutes.
Ad-hoc analytics with PySpark
Data scientists explore large datasets in Jupyter notebooks using familiar pandas syntax.
Fraud detection streaming pipeline
Ingests transaction streams, applies ML models, and alerts in near-real time.
Social network graph analysis
Computes PageRank and community detection on billions of edges using GraphX.
Use ./bin/spark-shell for Scala or ./bin/pyspark for Python; both connect to a local or configured cluster.
Scala, Java, Python, and (deprecated) R APIs are provided out of the box.
Yes, Spark integrates with Hadoop storage and can be launched on YARN, Mesos, or Kubernetes alongside Hadoop services.
Use ./bin/run-example or spark-submit with the MASTER environment variable set to spark://host:port, yarn, or local[N].
Project at a glance
ActiveLast synced 4 days ago
