Geological reconstructions of global ocean chemistry and atmospheric oxygen concentrations over Earth history commonly rely on the abundance and stable isotopic composition (δ34S) of sulfur-bearing compounds. Carbonate-associated sulfate (CAS), sulfate bound within a calcium carbonate mineral matrix, is among the most commonly interrogated sulfur mineral phases. However, recent work has revealed variability in δ34SCAS values that cannot be explained by evolution of the marine sulfate reservoir, challenging the common interpretation that CAS is inherently a high-fidelity record of seawater sulfate. To investigate the source of this inconsistency, we used X-ray spectromicroscopy to map the micron-scale distribution of S-bearing sedimentary phases in Ordovician-aged (ca. 444 Ma) shallow marine carbonates from Anticosti Island, Québec, Canada. Clear differences in the abundance of CAS are observed between cements and fossils, suggesting that variance in bulk-rock data could be a consequence of component mixing and that coupled synchrotron-petrographic screening can identify the carbonate components that are most likely to retain primary CAS. Furthermore, we observe multiple, distinct forms of sulfate (both inorganic and organic). Differences in these forms among fossil clades could provide new insights into biomineralization mechanisms in extinct organisms.