The Central Asian Orogenic Belt has long been considered the largest Phanerozoic accretionary orogen in the world; it developed through the subduction and final closure of the Paleo–Asian Ocean. However, the architecture and duration of the accretionary orogenesis of the Central Asian Orogenic Belt are still controversial despite decades of investigation. In this study, we present field, compositional, and stratigraphically controlled detrital zircon geochronological data for late Paleozoic sedimentary rocks from the Alxa Tectonic Belt to constrain their provenance, tectonic setting, and the overall tectonic configuration of the southern Central Asian Orogenic Belt. A Devonian sample yields a unimodal age peak (ca. 424 Ma) and broad late Mesoproterozoic ages. A Carboniferous sample has Early Silurian (ca. 438 Ma) and Late Devonian (ca. 382 Ma) peaks along with Neoproterozoic to Archean ages. The Permian samples are dominated by Ordovician–Devonian and Carboniferous–Permian ages. They yield maximum depositional ages ranging from ca. 291 Ma to 248 Ma and contain abundant zircon ages that are close to their depositional ages. These data reveal Ordovician–Silurian and Carboniferous–Permian magmatic flare-ups separated by a Devonian magmatic lull in the southern Central Asian Orogenic Belt. The arc terranes in southern Mongolia, central Beishan, and northern Alxa provided major detritus for the late Paleozoic sediments. An abrupt shift of zircon εHf(t) values at ca. 400 Ma reveals significant late Paleozoic crustal growth and excludes southern Alxa as a source. Oceanic basins prevented detritus from southern Alxa from reaching northern Alxa during Permian–Early Triassic time. A geological and provenance comparison of Permian basins in the southern Central Asian Orogenic Belt reveals the existence of two separate forearcs ascribed to bipolar subduction of the Paleo–Asian Ocean. Combined with recent paleomagnetic data, this leads us to advocate for an archipelago-style accretionary process induced by subduction retreat for the late Paleozoic tectonic evolution of the southern Central Asian Orogenic Belt, which continued into Late Permian–Early Triassic.