Design and Performance Comparison of Batch and Streaming Data Processing Pipelines
2026 (Engelska)Självständigt arbete på grundnivå (kandidatexamen), 10 poäng / 15 hp
Studentuppsats (Examensarbete)Alternativ titel
Design- och prestandajämförelse av batch- och strömbaserade databehandlingspipelines (Svenska)
Abstract [en]
Data pipelines are a core component of distributed systems, where large volumes of data must be processed efficiently, reliably, and at scale. The two commonly used paradigms are batch processing, which handles data in bounded jobs, and stream processing, which handles data continuously as it arrives. Despite both being widely used, little empirical research compares them as deployed pipelines under controlled and comparable conditions. This thesis addresses the gap by implementing a batch pipeline using Apache Spark, and a streaming pipeline with Apache Flink and Apache Kafka, then comparing their performance in a controlled experiment. The pipelines run an identical hourly aggregation over NYC Yellow Taxi records across three dataset sizes and three parallelism levels, with each condition repeated five times on two machines. The evaluation measures throughput, latency, CPU utilisa tion, memory usage, and duration. The batch pipeline reached higher throughput at every dataset size, and the gap widened as the data grew. The streaming pipeline held stable throughput and resource utilisation, at the cost of longer duration. The results also challenge two common assumptions, implying that under this workload batch scaled with the available cores, while streaming throughput remained limited by its single-threaded producer rather than by Flink, and batch was not generally the heavier paradigm.Therefore, the choice between the architectures is conditional. Batch is more suitable for workloads where high throughput is the priority, while streaming is preferable where continuous processing and timely reactions to incoming data are required. The final decision depends on the workload, the latency requirements, and the operational constraints.
Ort, förlag, år, upplaga, sidor
2026. , s. 45
Nyckelord [en]
data processing pipelines, distributed systems, batch architecture, stream architecture
Nationell ämneskategori
Datavetenskap (datalogi)
Identifikatorer
URN: urn:nbn:se:lnu:diva-149172OAI: oai:DiVA.org:lnu-149172DiVA, id: diva2:2094053
Utbildningsprogram
Programvaruteknik, 180 hp
Handledare
Examinatorer
2026-09-012026-08-202026-09-01Bibliografiskt granskad