On Earth, carbon sequestration in geologic units plays an important role in the carbon cycle, scrubbing CO2 from the atmosphere for long-term storage. While carbonate is identified in low abundances within the dust and soils of Mars, at <1 wt% in select meteorites, and in limited outcrops, no massive carbonate rock reservoir on Mars has been identified to date. Here, we investigate the largest exposed carbonate-bearing rock unit, the Nili Fossae plains, combining spectral, thermophysical, and morphological analyses to evaluate the timing and carbon sequestration potential of rocks on Mars. We find that the olivine-enriched (∼20%–25%) basalts have been altered, by low-temperature in situ carbonation processes, to at most ∼20% Fe-Mg carbonate, thus limiting carbon sequestration in the Nili Fossae region to ∼0.25–12 mbar of CO2 during the late Noachian–early Hesperian, before or concurrent with valley network formation. While this is large compared to modern-day CO2 reservoirs, the lack of additional, comparably sized post–late Noachian carbonate-bearing deposits on Mars indicates ineffective carbon sequestration in rock units over the past ∼3.7 b.y. This implies a thin atmosphere (≲500 mbar) during valley network formation, extensive post-Noachian atmospheric loss to space, or diffuse, deep sequestration by a yet-to-be understood process. In stark contrast to Earth’s biologically mediated crust:atmosphere carbon reservoir ratio of ∼104–105, Mars’ ratio is a mere ∼10–103, even if buried pre-Noachian crust holds multiple bars.