Recent developments in artificial intelligence (AI) and its ubiquitous presence in society have caused debate about its possible implications for individuals and society, not least in the domain of education. An ongoing discussion about what knowledge related to artificial intelligence is needed to navigate and take part in an increasingly data-driven society is currently shaping and defining the research area of AI literacy. Previous research has focused on identifying what pupils should know, but more needs to be discovered about what pupils already know about AI. To this end, this study explored the conceptualization of AI among more than 120 Swedish K-12 students and investigated the effectiveness of AI literacy interventions in enhancing their critical thinking and ethical awareness. The students displayed diverse and often inaccurate preconceptions of AI, often associating AI with physical computing and robotics. Although the pupils seemed to engage in and understand particular ML concepts at the time of intervention, it was a more difficult task for them to transfer this knowledge to different contexts in which such ML concepts are embedded.