The study analyses the life cycle cost efficiency and primary energy use to retrofit a multi-storey residential building to an annual final energy use of 50 and 30 kWh per heated floor area, respectively, corresponding to two Swedish passive house standards. We compare the net present value (NPV) of the net primary energy savings of the retrofitting options and the cost of them. The retrofit includes additional insulation to basement walls, exterior walls, and attic floor, new windows, efficient water taps and ventilation heat recovery. We assume the use of different building materials, including glass wool, rock wool and wood fibre for thermal insulation, and aluminium, brick and wood for cladding. We also analyse the effects of various energy scenarios on the primary energy savings and costs of the building retrofit. The heat supply is from bio-based or electric heat pump district heating. The electricity is from standalone fossil gas (100%) or bio (30%) and wind electricity (70%) plants. The biomass cost is based on current or 60% higher prices. Retrofit to 50 kWh per heated floor area is cost efficient while the 30 kWh level is close to be cost efficient for higher biomass prices. The primary energy use to produce the retrofit options is much smaller in all cases than the operation primary energy savings of them. The cost for different retrofitting material options is rather similar while the production primary energy use is much lower for wood based materials.