Evaluating trade-offs on different KV cache quantization levels and RAM constraints on edge AI
2026 (engelsk)Independent thesis Basic level (degree of Bachelor), 10 poäng / 15 hp
OppgaveAlternativ tittel
Utvärdering av avvägningar mellan olika nivåer av KV-cachekvantisering och RAM-begränsningar inom Edge AI (svensk)
Abstract [en]
Artificial Intelligence (AI) is rising in popularity day by day. More people are looking for a more local alternative instead of the traditional cloud-based AI models. A lot of these cloud-based AI services are not hosted locally; more often than not, they are hosted in a completely different country. This thesis looked at one of the best locally hosted alternatives, this being edge AI models. Edge AI runs locally on all sorts of Internet of Things (IoT) but with the recent rise in hardware components pricing, could outdated devices be capable of running these models? This thesis investigates how different KV-cache formats interact with RAM constraints in a CPU-based local inference environment. Controlled experiments were conducted in which different quantization levels were tested with different RAM constraints. The results show that the effect of KV-cache format on latency depended on the available RAM. They also show that the calculated storage requirements of the cache are not directly reflected in total runtime memory behaviour. There was no difference in answer correctness with all runs getting 11/12 answers correct while failing the same question. These results indicate that the performance effects of KV-cache quantization depend on the available RAM and runtime environment.
sted, utgiver, år, opplag, sider
2026.
Emneord [en]
Edge-AI, KV cache, AI (Artificial Intelligence), Quantization, RAM (Random Access Memory)
HSV kategori
Identifikatorer
URN: urn:nbn:se:lnu:diva-149770OAI: oai:DiVA.org:lnu-149770DiVA, id: diva2:2105278
Utdanningsprogram
Software Technology Programme, 180 credits
Presentation
2026-08-10, D2273V, Vejdes plats 6, 352 52, Växjö, 09:00 (engelsk)
Veileder
Examiner
2026-09-302026-09-252026-09-30bibliografisk kontrollert