Under climate warming conditions, storage and conversion of soil inorganic carbon (SIC) play an important role in regulating soil carbon (C) dynamics and atmospheric CO2 content in arid and semi-arid areas. Carbonate formation in alkaline soil can fix a large amount of C in the form of inorganic C, resulting in soil C sink and potentially slowing global warming trends. Therefore, understanding the driving factors affecting carbonate mineral formation can help better predict future climate change. Till date, most studies have focused on abiotic drivers (climate and soil), whereas a few examined the effects of biotic drivers on carbonate formation and SIC stock. In this study, SIC, calcite content, and soil microbial communities were analyzed in three soil layers (0–5 cm, 20–30 cm, and 50–60 cm) on the Beiluhe Basin of Tibetan Plateau. Results revealed that in arid and semi-arid areas, SIC and soil calcite content did not exhibit significant differences among the three soil layers; however, the main factors affecting the calcite content in different soil layers are different. In the topsoil (0–5 cm), the most important predictor of calcite content was soil water content. In the subsoil layers 20–30 cm and 50–60 cm, the ratio of bacterial biomass to fungal biomass (B/F) and soil silt content, respectively, had larger contributions to the variation of calcite content than the other factors. Plagioclase provided a site for microbial colonization, whereas Ca2+ contributed in bacteria-mediated calcite formation. This study aims to highlight the importance of soil microorganisms in managing soil calcite content and reveals preliminary results on bacteria-mediated conversion of organic to inorganic C.