Natural colonization of abandoned agricultural soils has been proposed as a cost-effective strategy for simultaneously mitigating the climate crisis, restoring ecological integrity, and promoting the re-establishment of native biodiversity. The success and speed at which forests develop in the presence of land-use legacies are highly variable, and empirical knowledge on the drivers of the initial phase of temperate forest colonization in a range of site conditions is lacking. We analyzed and compared drivers of the initial three decades of natural forest colonization in 33 afforestation sites laid out between 1990 and 2018, in Denmark. We analyzed how the age of colonization, size of the area, soil type, topography, and abundance of neighboring woody vegetation influenced woody vegetation cover, species richness, and variation in vegetation structure of the colonizing woody species. We found that woody vegetation cover, species richness, and structural variation increased significantly with time since abandonment. Woody vegetation cover was significantly higher in sites with abundant neighboring vegetation and on loamy soils compared to sand or clay. Woody vegetation cover tended to be higher in sites with sloping terrain. Species richness increased significantly in sites with loamy or clay soil and sites with sloping terrain. After accounting for the other drivers, none of the elements were influenced by the size of the area. The results suggest that time since abandonment, neighboring mature woody vegetation, soil and topography are key drivers of the different elements of colonization. This knowledge can be used to prioritize areas for natural colonization and to improve active afforestation methods, especially in new forests where the main focus is on nature quality and biodiversity.