This article presents a comprehensive analysis of the influence of the neighborhood-level presence of visible minorities on xenophobic party support. Drawing on previous research on support for xenophobic political parties in Europe, we explore variations in electoral preferences for the Sweden Democrats. We examine relationships between the presence of visible minorities at the neighborhood level, controlling for the level of unemployment and education both in terms of potentially additive and interactive effects. We find that increased exposure of visible minorities unambiguously leads to an increase in xenophobic voting if the district level of unemployment is high. Where unemployment is low the effect of increased exposure may instead lead to reduced xenophobic party support, due to complex interaction effects involving aggregate education level. The results are discussed in the light of possible public policy measures to combat unemployment in multicultural democracies.