The concentrations and ecological risk of six widespread heavy metals (Cu, Cr, Ni, Zn, Cd and Pb) were investigated and evaluated in sediments from both urban and rural rivers in a northeast city of China. The decreasing trend of the average concentration of heavy metals was Zn > Cr > Cu > Pb > Ni > Cd in Majiagou River (urban) and was Zn > Cr > Pb > Cu > Ni > Cd in Yunliang River (rural). The results showed that the concentrations of Cd and Zn were significantly elevated compared to the environmental background value (p < 0.05). Half of all sampling locations were deemed 'contaminated' as defined by the improved Nemerow pollution index (P-N' > 1.0). Applying the potential ecological risk index (RI) indicated a 'high ecological risk' for both rivers, with Cd accounting for more than 80% in both cases. Source apportionment indicated a significant correlation between Cd and Zn in sediments (R = 0.997, p < 0.01) in Yunliang River, suggesting that agricultural activities could be the major sources. Conversely, industrial production, coal burning, natural sources and traffic emissions are likely to be the main pollution sources for heavy metals in Majiagou River. This study has improved our understanding of how human activities, industrial production, and agricultural production influence heavy metal pollution in urban and rural rivers, and it provides a further weight of evidence for the linkages between different pollutants and resulting levels of heavy metals in riverine sediments.