lnu.sePublications
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Design and Performance Comparison of Batch and Streaming Data Processing Pipelines
Linnaeus University, Faculty of Technology, Department of computer science and media technology.
Linnaeus University, Faculty of Technology, Department of computer science and media technology.
2026 (English)Independent thesis Basic level (degree of Bachelor), 10 credits / 15 HE creditsStudent thesisAlternative title
Design- och prestandajämförelse av batch- och strömbaserade databehandlingspipelines (Swedish)
Abstract [en]

Data pipelines are a core component of distributed systems, where large volumes of data must be processed efficiently, reliably, and at scale. The two commonly used paradigms are batch processing, which handles data in bounded jobs, and stream processing, which handles data continuously as it arrives. Despite both being widely used, little empirical research compares them as deployed pipelines under controlled and comparable conditions. This thesis addresses the gap by implementing a batch pipeline using Apache Spark, and a streaming pipeline with Apache Flink and Apache Kafka, then comparing their performance in a controlled experiment. The pipelines run an identical hourly aggregation over NYC Yellow Taxi records across three dataset sizes and three parallelism levels, with each condition repeated five times on two machines. The evaluation measures throughput, latency, CPU utilisa tion, memory usage, and duration. The batch pipeline reached higher throughput at every dataset size, and the gap widened as the data grew. The streaming pipeline held stable throughput and resource utilisation, at the cost of longer duration. The results also challenge two common assumptions, implying that under this workload batch scaled with the available cores, while streaming throughput remained limited by its single-threaded producer rather than by Flink, and batch was not generally the heavier paradigm.Therefore, the choice between the architectures is conditional. Batch is more suitable for workloads where high throughput is the priority, while streaming is preferable where continuous processing and timely reactions to incoming data are required. The final decision depends on the workload, the latency requirements, and the operational constraints.

Place, publisher, year, edition, pages
2026. , p. 45
Keywords [en]
data processing pipelines, distributed systems, batch architecture, stream architecture
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:lnu:diva-149172OAI: oai:DiVA.org:lnu-149172DiVA, id: diva2:2094053
Educational program
Software Technology Programme, 180 credits
Supervisors
Examiners
Available from: 2026-09-01 Created: 2026-08-20 Last updated: 2026-09-01Bibliographically approved

Open Access in DiVA

fulltext(7287 kB)15 downloads
File information
File name FULLTEXT01.pdfFile size 7287 kBChecksum SHA-512
bd2481771532253694d9f0f48912473364b3307392d5c0a0e7d9c0b3b8a0c9c261f95096eed5e2f379ca57c9f794cde5751e9acb92a437524ffd09094ab78983
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Aleksandrova, AleksandraAmbroziak, Klaudia
By organisation
Department of computer science and media technology
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 47 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf