Ghost forests, consisting of dead trees adjacent to marshes, are a striking feature of low-lying coastal and estuarine landscapes, and they represent the migration of coastal ecosystems with relative sea-level rise (RSLR). Although ghost forests have been observed along many coastal margins, rates of ecosystem change and their dependence on RSLR remain poorly constrained. Here, we reconstructed forest retreat rates using sediment coring and historical imagery at five sites along the Mid-Atlantic coast of the United States, a hotspot for accelerated RSLR. We found that the elevation of the marsh-forest boundary generally increased with RSLR over the past 2000 yr, and that retreat accelerated concurrently with the late 19th century acceleration in global sea level. Lateral retreat rates increased through time for most sampling intervals over the past 150 yr, and modern lateral retreat rates are 2 to 14 times faster than pre-industrial rates at all sites. Substantial deviations between RSLR and forest response are consistent with previous observations that episodic disturbance facilitates the mortality of adult trees. Nevertheless, our work suggests that RSLR is the primary determinant of coastal forest extent, and that ghost forests represent a direct and prominent visual indicator of climate change.