This study investigated heritability of stem and wood traits to improve Swedish silver birch (Betula pendula Roth.) through breeding. Birch is 12% of Sweden’s forest area but mainly used for low value pulp or firewood. This paper applied non-destructive test (NDT) methods, and estimated traits’ heritability (h2), to help breed birch for high value solid wood products. Two trials of 22 families were assessed at age 19 for stem diameter (DBH), stem straightness, rough brown bark height (BH), grain angle (GA), Pilodyn penetration depth (Pilo) and acoustic velocity (AV). X-ray densitometry was performed on a subsample of radial cores taken at 1.3 m from the ground to get an average benchmark density. The h2 values were moderate for GA (0.20 and 0.21) and Pilo (0.53 and 0.48) at the two sites, but the h2 values for AV were low (0.05 and 0.30). There were moderate genotypic correlations between BH and DBH (0.51–0.54). There were low genotypic and phenotypic correlations between NDT measurements and other traits so including NDT in birch breeding efforts should not inadvertently reduce size, stem or wood quality. The high genetic correlations between sites suggest that GA, Pilo and AV values were determined more by genotype than by environment.