Recent studies have highlighted the dynamic behavior of marine-terminating outlet glaciers over decadal time scales, linked to both atmospheric and oceanic warming. This helps explain episodes of nearly synchronous flow acceleration, thinning, and retreat, but nonclimatic factors such as changes in fjord width and depth, can also induce rapid recession. There is support for these topographic controls on glacier retreat, but there are few long-term records to assess their significance across a population of glaciers over millennial time scales. Here we present retreat chronologies along with topographic data for eight major outlet glaciers that underwent similar climatic forcing during deglaciation of the Fennoscandian Ice Sheet (ca. 18–10 ka). Retreat rates averaged over several millennia (∼30 m a–1) are less than half those recently observed on modern-day outlet glaciers (>100 m a–1), but deglaciation was punctuated by episodes of more rapid retreat (to ∼150 m a–1) and readvances. It is significant that phases of rapid retreat were not synchronous between glaciers and most occurred regardless of any obvious atmospheric warming. We interpret this to reflect the complex interplay between external forcing and both topographic (e.g., bathymetry, width) and glaciological factors (e.g., ice catchments) that evolve through time, but conclude that basal overdeepenings in wide fjords induce episodes of rapid retreat (>100 m a–1), further exacerbated by their greater susceptibility to oceanic warming. This complicates attempts to predict the centennial-scale trajectory of outlet glaciers and suggests that modeling the interaction between neighboring catchments and the accurate description of subglacial topography beneath them are priorities for future work.