The tectonic affinity of the terranes and microcontinents within the Central Asian Orogenic Belt (CAOB) remains controversial. The Altai-Mongolian terrane (AMT), as a representative tectonic unit in the Mongolian collage, plays a vital role in reconstructing evolution history of the CAOB. The well-preserved early Paleozoic sedimentary sequence covering in this terrane could be considered as a fingerprint to track its provenance and tectonic affinity. Here, we present new whole-rock geochemistry, detrital zircon U-Pb dating, and Hf isotopic analysis for the metasedimentary rocks from the Mongolian Altai in order to shed new light on the tectonic affinity of the AMT. The youngest detrital zircon ages and the regional intrusions constrain the depositional time of the Mongolian Altai sequence to between Late Silurian and Early Devonian, which is consistent with the Habahe group in the western Chinese Altai. The features of whole-rock geochemistry and the cumulative distribution curves of the detrital zircon age spectra indicate that the Mongolian Altai sequence was probably deposited in an active continental setting during early Paleozoic. The zircon age spectra of our samples are all characterized by a main age group in the early Cambrian (peak at 541 Ma, 522 Ma, 506 Ma and 496 Ma, respectively), subdominant age populations during the Tonian, as well as rare older zircons. The nearby Lake Zone of Ikh-Mongol Arc most likely provided plenty of early Paleozoic materials, the subdominant Neoproterozoic detrital zircons could be supplied by the felsic intrusions along the western margin of the Tuva-Mongol microcontinent, and the sparse older zircons may be derived from its basement material. The Precambrian age distribution of the AMT is quite similar to both the Tarim and Siberia cratons, but the Siberia Craton displays a closer resemblance in Hf isotopic composition with the AMT. Thus, we believe that the Siberia Craton contains a closer tectonic affinity with the AMT, and that the Tuva-Mongol microcontinent possibly rifted from the western margin of this craton after the Tonian. To the south of the AMT, recent studies indicated the Yili and Central Tianshan blocks in the Kazakhstan collage of the western CAOB likely have a closer affinity with Gondwana. Therefore, the microcontinents in the CAOB most likely derived bilaterally from both the Siberia Craton and the Gondwana supercontinent. Moreover, our Hf isotopic compositions indicate two significant continental growth events in the Tonian and early Paleozoic, respectively.