This article analyses the images of the criminal that Swedish prison chaplains developed during the 19^th century. Following Michel Foucault, such images and relations are conceptualized as pastoral power. In the 19^th century prison chaplains occupied a very important position within prisons, due partly to their status as clergymen, but also to their increasing influence on debates about crime and punishment. This article argues that in many respects prison chaplains came to develop a more sophisticated understanding of the criminal and the complexities of criminality, compared to many early psychiatrists and criminal anthropologists. This understanding was primarily the result of knowledge-gathering which took place within the penitentiaries. In the mid-nineteenth century prison chaplains generally depicted crime as morally determined and as caused by the lack of knowledge and insights into the truths of Christianity. However, during the last quarter of the century, references to social and structural factors were increasingly included in broader and more heterogeneous explanations for crime. However, if non-moral factors played such a central role as the prison chaplains gradually came to assign to them, then their own authority as experts on crime as well as the status of religion within the prison system was undermined.