lnu.sePublikationer
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Design and Performance Comparison of Batch and Streaming Data Processing Pipelines
Linnéuniversitetet, Fakulteten för teknik (FTK), Institutionen för datavetenskap och medieteknik (DM).
Linnéuniversitetet, Fakulteten för teknik (FTK), Institutionen för datavetenskap och medieteknik (DM).
2026 (Engelska)Självständigt arbete på grundnivå (kandidatexamen), 10 poäng / 15 hpStudentuppsats (Examensarbete)Alternativ titel
Design- och prestandajämförelse av batch- och strömbaserade databehandlingspipelines (Svenska)
Abstract [en]

Data pipelines are a core component of distributed systems, where large volumes of data must be processed efficiently, reliably, and at scale. The two commonly used paradigms are batch processing, which handles data in bounded jobs, and stream processing, which handles data continuously as it arrives. Despite both being widely used, little empirical research compares them as deployed pipelines under controlled and comparable conditions. This thesis addresses the gap by implementing a batch pipeline using Apache Spark, and a streaming pipeline with Apache Flink and Apache Kafka, then comparing their performance in a controlled experiment. The pipelines run an identical hourly aggregation over NYC Yellow Taxi records across three dataset sizes and three parallelism levels, with each condition repeated five times on two machines. The evaluation measures throughput, latency, CPU utilisa tion, memory usage, and duration. The batch pipeline reached higher throughput at every dataset size, and the gap widened as the data grew. The streaming pipeline held stable throughput and resource utilisation, at the cost of longer duration. The results also challenge two common assumptions, implying that under this workload batch scaled with the available cores, while streaming throughput remained limited by its single-threaded producer rather than by Flink, and batch was not generally the heavier paradigm.Therefore, the choice between the architectures is conditional. Batch is more suitable for workloads where high throughput is the priority, while streaming is preferable where continuous processing and timely reactions to incoming data are required. The final decision depends on the workload, the latency requirements, and the operational constraints.

Ort, förlag, år, upplaga, sidor
2026. , s. 45
Nyckelord [en]
data processing pipelines, distributed systems, batch architecture, stream architecture
Nationell ämneskategori
Datavetenskap (datalogi)
Identifikatorer
URN: urn:nbn:se:lnu:diva-149172OAI: oai:DiVA.org:lnu-149172DiVA, id: diva2:2094053
Utbildningsprogram
Programvaruteknik, 180 hp
Handledare
Examinatorer
Tillgänglig från: 2026-09-01 Skapad: 2026-08-20 Senast uppdaterad: 2026-09-01Bibliografiskt granskad

Open Access i DiVA

fulltext(7287 kB)29 nedladdningar
Filinformation
Filnamn FULLTEXT01.pdfFilstorlek 7287 kBChecksumma SHA-512
bd2481771532253694d9f0f48912473364b3307392d5c0a0e7d9c0b3b8a0c9c261f95096eed5e2f379ca57c9f794cde5751e9acb92a437524ffd09094ab78983
Typ fulltextMimetyp application/pdf

Sök vidare i DiVA

Av författaren/redaktören
Aleksandrova, AleksandraAmbroziak, Klaudia
Av organisationen
Institutionen för datavetenskap och medieteknik (DM)
Datavetenskap (datalogi)

Sök vidare utanför DiVA

GoogleGoogle Scholar
Antalet nedladdningar är summan av nedladdningar för alla fulltexter. Det kan inkludera t.ex tidigare versioner som nu inte längre är tillgängliga.

urn-nbn

Altmetricpoäng

urn-nbn
Totalt: 109 träffar
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf