Calcium carbonate (CaCO3) and particularly its stable phase, calcite, is of great geological significance in the deep carbon cycle since CaCO3 from biomineralized shells and corals form sedimentary rocks. Calcite also attracts attention in medical science and pharmacy as a primary or intermediate component in biomaterials because it possesses excellent biocompatibility along with suitable physicochemical properties. Calcite blocks have already been used during surgical procedures as a bone substitute for reconstructing bone defects formed by diseases and injury. When producing CaCO3 biomaterials and bioceramics, in particular, in vivo control of the size and polymorphic nature of CaCO3 is required. In this study, we investigated the effects of PO4 on calcite formation during the phase conversion of calcium sulfate anhydrate (CaSO4, CSA), which is sometimes used as a starting material for bone substitutes because of its suitable setting ability. CSA powder was immersed in 2 mol/L Na2CO3 solution containing a range of PO4 concentrations (0–60 mmol/L) at 40 °C for 3 days. The treated samples were investigated by X-ray diffraction, Fourier-transform infrared spectroscopy, X-ray fluorescence spectroscopy, and thermal analysis. In addition, the fine structures of the treated samples were observed by field-emission scanning electron microscopy, and the specific surface area was measured. We found that PO4, which is universally present in vivo, can modulate the calcite crystal size during calcite formation. A fluorescence study and calcite crystal growth experiments indicated that PO4 adsorbs tightly onto the surface of calcite, inhibiting crystal growth. In the presence of high PO4 concentrations, vaterite is formed along with calcite, and the appearance and stability of the CaCO3 polymorphs can be controlled by adjusting the PO4 concentration. These findings have implications for medical science and pharmacology, along with mineralogy and geochemistry.