Soil organic carbon management is a nature-based carbon dioxide removal technology at the same time contributing to soil health and agricultural productivity. The soil science communities are refuting the traditional assumptions of the nature of soil organic matter (SOM) as based on ‘humic substances’ that are operationally-defined and have not been observed by contemporary, in situ spectromicroscopic techniques. Instead, new theories suggest that the interactions between molecular diversity of organic compounds, their spatial heterogeneity and temporal variability controls the formation and persistence of SOM. A mechanistic understanding of these processes occurring within organo-mineral and organo-organic assemblages requires non-invasive techniques that minimize any disturbance to the physical and chemical integrity of the sample. Here, we present a theory-driven review where a combination of in situ methods serve as potential solutions to better understand the persistence and dynamics of SOM and its effects on nutrient distribution at a micro- and nano-scale. We explore underlying theories in light of advances in available methodologies, their historical development and future opportunities. Examples of interdisciplinary approaches that have been utilized in other areas of science but not in soils offer both deductive and inductive analytical opportunities. We show how different conceptual methods across scales inform each other, and how important and indispensable high-resolution investigations are to resolving next-generation questions.