Building renovation is considered a key strategy to facilitating the transition towards a renewable energy system, mitigating climate change and addressing energy poverty. However, the rate of buildings undergoing deep energy retrofit in Europe is below 1% per year. The study analyses the life cycle cost and primary energy impacts to retrofit a residential building to an annual final energy use of 50 and 30 kWh/m2, respectively, corresponding to two Swedish passive house standards. The retrofit includes the thermal improvement of the building envelope, and the increased efficiency of water taps and ventilation system. We assume the use of different retrofit materials. The study also analyses the effects of various energy scenarios for district heating and electricity on the primary energy savings and costs of the building retrofit. Finally, we compare the economic net present value (NPV) of the net primary energy savings due to the retrofit options and the cost of them. Retrofit to 50 kWh/m2 is cost efficient while the 30 kWh/m2 level is close to be cost efficient for higher biomass prices. The primary energy use to produce the retrofit options is much smaller in all cases than the operation primary energy savings of them. The cost of different retrofit material versions is similar while the production primary energy use is much lower for wood based materials. Different electricity production scenarios affect the net primary energy savings but are marginal in terms of costs.