BigDataFr recommends: Kera: A Unified Storage and Ingestion Architecture for Efficient Stream Processing
Abstract
[…] Big Data applications are rapidly moving from a batch-oriented execution to a real-time model in order to extract value from the streams of data just as fast as they arrive. Such stream-based applications need to immediately ingest and analyze data and in many use cases combine live (i.e., real-time streams) and archived data in order to extract better insights.
Current streaming architectures are designed with distinct components for ingestion (e.g., Kafka) and storage (e.g., HDFS) of stream data. Unfortunately, this separation is becoming an overhead especially when data needs to be archived for later analysis (i.e., near real-time): in such use cases, stream data has to be written twice to disk and may pass twice over high latency networks. Moreover, current ingestion mechanisms offer no support for searching the acquired streams in real time, an important requirement to promptly react to fast data. In this paper we describe the design of Kera: a unified storage and ingestion architecture that could better serve the specific needs of stream processing. […]
Read paper
By Ovidiu-Cristian Marcu 1, Alexandru Costan 1, 2 Gabriel Antoniu 1, María S. Pérez-Hernández 3
Source: hal-archives-ouvertes.fr
1 KerData – Scalable Storage for Clouds and Beyond
Inria Rennes – Bretagne Atlantique , IRISA-D1 – SYSTÈMES LARGE ÉCHELLE
2 INSA Rennes – Institut National des Sciences Appliquées – Rennes
3 Universidad Politécnica de Madrid