We analyze conventional retrofit building materials, aluminum, rock, and glass wool materials and compared such materials with wood-based materials to understand the lifecycle primary energy implications of moving from non-renewable to wood-based materials. We calculate cost optimum retrofit measures for a multi-apartment building in a lifecycle perspective, and lifecycle primary energy savings of each optimized measure. The retrofit measures consist of the thermal improvement of windows with varied frame materials, as well as extra insulation of attic floor, basement walls, and external walls with varied insulation materials. The most renewable-based heat supply is from a bioenergy-based district heating (DH) system. We use the marginal cost difference method to calculate cost-optimized retrofit measures. The net present value of energy cost savings of each measure with a varied energy performance is calculated and then compared with the calculated retrofit cost to identify the cost optimum of each measure. In a sensitivity analysis, we analyze the cost optimum retrofit measures under different economic and DH supply scenarios. The retrofit costs and primary energy savings vary somewhat between non-renewable and wood-based retrofit measures but do not influence the cost optimum levels significantly, as the economic parameters do. The lifecycle primary use of wood fiber insulation is about 76% and 80% less than for glass wool and rock wool, respectively. A small-scale DH system gives higher primary energy and cost savings compared to larger DH systems. The optimum final energy savings, in one of the economic scenarios, are close to meeting the requirements in one of the Swedish passive house standards.