The weathering of carbonate rocks plays a critical role in the evolution of landscapes, the erosion of buildings and monuments, and the global-scale shifting of carbon from the atmosphere to the ocean. Chemical dissolution is often assumed to govern the rates of weathering of carbonate rocks, although some studies have suggested that mechanical erosion could also play an important role. Quantifying the rates of the different processes has proved challenging, in part due to the high degree of variability encountered across different scales in both field and laboratory conditions. To constrain the rates and mechanisms controlling long-term limestone weathering, we analyze a lidar scan of the Western Wall, a Roman-period edifice located in Jerusalem. We find that extreme erosion rates in fine-grained micritic limestone blocks are as much as two orders of magnitude higher than the average rates estimated for coarse-grained limestone blocks at the same site. Atomic force microscope imaging of dissolving micritic limestone suggests that these elevated reaction rates are likely to be the result of rapid dissolution along micron-scale grain boundaries, followed by mechanical detachment of tiny particles from the surface. Our analysis indicates that such grain detachment could be the dominant erosional mode for fine-grained carbonate rocks in many regions on Earth.