The stability of magnetic skyrmions against thermal fluctuations and external perturbations is investigated within the framework of harmonic transition state theory for magnetic degrees of freedom. The influence of confined geometry and atomic scale non-magnetic defects on the skyrmion lifetime is estimated. It is shown that a skyrmion on a track has lower activation energy for annihilation and higher energy for nucleation if the size of the skyrmion is comparable with the width of the track. Two mechanisms of skyrmion annihilation are considered: inside the track and escape through the boundary. For both mechanisms, the dependence of activation energy on the track width is calculated. Non-magnetic defects are found to localize skyrmions in their neighborhood and strongly decrease the activation energy for creation and annihilation. This is in agreement with experimental measurements that have found nucleation of skyrmions in presence of spin-polarized current preferably occurring near structural defects.