Design and Performance Comparison of Batch and Streaming Data Processing Pipelines
2026 (English)Independent thesis Basic level (degree of Bachelor), 10 credits / 15 HE credits
Student thesisAlternative title
Design- och prestandajämförelse av batch- och strömbaserade databehandlingspipelines (Swedish)
Abstract [en]
Data pipelines are a core component of distributed systems, where large volumes of data must be processed efficiently, reliably, and at scale. The two commonly used paradigms are batch processing, which handles data in bounded jobs, and stream processing, which handles data continuously as it arrives. Despite both being widely used, little empirical research compares them as deployed pipelines under controlled and comparable conditions. This thesis addresses the gap by implementing a batch pipeline using Apache Spark, and a streaming pipeline with Apache Flink and Apache Kafka, then comparing their performance in a controlled experiment. The pipelines run an identical hourly aggregation over NYC Yellow Taxi records across three dataset sizes and three parallelism levels, with each condition repeated five times on two machines. The evaluation measures throughput, latency, CPU utilisa tion, memory usage, and duration. The batch pipeline reached higher throughput at every dataset size, and the gap widened as the data grew. The streaming pipeline held stable throughput and resource utilisation, at the cost of longer duration. The results also challenge two common assumptions, implying that under this workload batch scaled with the available cores, while streaming throughput remained limited by its single-threaded producer rather than by Flink, and batch was not generally the heavier paradigm.Therefore, the choice between the architectures is conditional. Batch is more suitable for workloads where high throughput is the priority, while streaming is preferable where continuous processing and timely reactions to incoming data are required. The final decision depends on the workload, the latency requirements, and the operational constraints.
Place, publisher, year, edition, pages
2026. , p. 45
Keywords [en]
data processing pipelines, distributed systems, batch architecture, stream architecture
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:lnu:diva-149172OAI: oai:DiVA.org:lnu-149172DiVA, id: diva2:2094053
Educational program
Software Technology Programme, 180 credits
Supervisors
Examiners
2026-09-012026-08-202026-09-01Bibliographically approved