lnu.sePublikasjoner
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Design and Performance Comparison of Batch and Streaming Data Processing Pipelines
Linnéuniversitetet, Fakulteten för teknik (FTK), Institutionen för datavetenskap och medieteknik (DM).
Linnéuniversitetet, Fakulteten för teknik (FTK), Institutionen för datavetenskap och medieteknik (DM).
2026 (engelsk)Independent thesis Basic level (degree of Bachelor), 10 poäng / 15 hpOppgaveAlternativ tittel
Design- och prestandajämförelse av batch- och strömbaserade databehandlingspipelines (svensk)
Abstract [en]

Data pipelines are a core component of distributed systems, where large volumes of data must be processed efficiently, reliably, and at scale. The two commonly used paradigms are batch processing, which handles data in bounded jobs, and stream processing, which handles data continuously as it arrives. Despite both being widely used, little empirical research compares them as deployed pipelines under controlled and comparable conditions. This thesis addresses the gap by implementing a batch pipeline using Apache Spark, and a streaming pipeline with Apache Flink and Apache Kafka, then comparing their performance in a controlled experiment. The pipelines run an identical hourly aggregation over NYC Yellow Taxi records across three dataset sizes and three parallelism levels, with each condition repeated five times on two machines. The evaluation measures throughput, latency, CPU utilisa tion, memory usage, and duration. The batch pipeline reached higher throughput at every dataset size, and the gap widened as the data grew. The streaming pipeline held stable throughput and resource utilisation, at the cost of longer duration. The results also challenge two common assumptions, implying that under this workload batch scaled with the available cores, while streaming throughput remained limited by its single-threaded producer rather than by Flink, and batch was not generally the heavier paradigm.Therefore, the choice between the architectures is conditional. Batch is more suitable for workloads where high throughput is the priority, while streaming is preferable where continuous processing and timely reactions to incoming data are required. The final decision depends on the workload, the latency requirements, and the operational constraints.

sted, utgiver, år, opplag, sider
2026. , s. 45
Emneord [en]
data processing pipelines, distributed systems, batch architecture, stream architecture
HSV kategori
Identifikatorer
URN: urn:nbn:se:lnu:diva-149172OAI: oai:DiVA.org:lnu-149172DiVA, id: diva2:2094053
Utdanningsprogram
Software Technology Programme, 180 credits
Veileder
Examiner
Tilgjengelig fra: 2026-09-01 Laget: 2026-08-20 Sist oppdatert: 2026-09-01bibliografisk kontrollert

Open Access i DiVA

fulltext(7287 kB)29 nedlastinger
Filinformasjon
Fil FULLTEXT01.pdfFilstørrelse 7287 kBChecksum SHA-512
bd2481771532253694d9f0f48912473364b3307392d5c0a0e7d9c0b3b8a0c9c261f95096eed5e2f379ca57c9f794cde5751e9acb92a437524ffd09094ab78983
Type fulltextMimetype application/pdf

Søk i DiVA

Av forfatter/redaktør
Aleksandrova, AleksandraAmbroziak, Klaudia
Av organisasjonen

Søk utenfor DiVA

GoogleGoogle Scholar
Antall nedlastinger er summen av alle nedlastinger av alle fulltekster. Det kan for eksempel være tidligere versjoner som er ikke lenger tilgjengelige

urn-nbn

Altmetric

urn-nbn
Totalt: 109 treff
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf