The aim of this study was to examine Swedish school pupils’ understanding of being positioned as a bullying victim in relation to participant roles that do not actively support the victim (i.e., bully, assistant, reinforcer and outsider roles). A social psychological and sociological theoretical framework was adopted. Data was collected through the use of focus group interviews, wherein pupils were asked for their perspectives on various participant roles depicted in a bullying vignette. A total of 74 pupils from 7 school classes in grades 5 and 6 participated. Data were analysed using Constructivist Grounded Theory methods. In their explanations about the actions of pupils in bullying situations, the interviewees pointed to the fear that pupils may feel in relation to the perceived risk of ending up in the victim position. Once positioned as the ‘victim’ it becomes difficult to change that position and the associated perceptions of others, highlighting a particularly vulnerable situation, entailing the risk of further bullying and stigmatization. The fear of being singled out was the main concern, suggested as the driving force behind why pupils bully, join in bullying, laugh at the situation or choose to remain passive. The main conclusions of the study highlight the importance of difference, social stigmatization and anxiety about social exclusion as intertwined, social processes in bullying. The findings challenge previous understandings of bullying as an act of harmful or aggressive intentionality, and instead highlight the relational and situational aspects of bullying.