During the Quaternary, periodic glaciations transformed mountain landscapes. However, characterizing the way in which mountain erosion changes between glacier- and river-dominated conditions has been elusive. Here, using samples from an offshore sedimentary core, we estimated the spatial distribution of erosion in the southern part of the Southern Alps of New Zealand during a full transition from the Last Glacial Maximum (LGM), ca. 20 ka, to the last millennium. Raman spectroscopy analyses of carbonaceous material revealed a marked change in the sediment provenance, which we interpreted to reflect the evolving erosion pattern of the mountain range. Over the Holocene, since at least ca. 9 ka, erosion was focused on the chlorite zone schist within the upper reaches of the valleys (>15–20 km distance from the mountain front), possibly dominated by large-magnitude landslides. During the last glaciation, the proportion of sediments from the biotite schist and higher-grade metamorphic rocks in the lower-lying areas closer to the mountain front (<15–20 km) was relatively higher, probably as a result of glacier carving. Our results suggest that glacier retreat during the last deglaciation caused an upstream localization of the high erosion rates, which is consistent with the snowline records in the Southern Alps and regional and global climate histories.