Hydrogen incorporation in olivine involves many OH defects, which will control the hydrogen solubility at mantle conditions. Several of these OH defects are identified from the investigation of forsterite (the olivine Mg end-member). We study here the effect of Fe2+, Fe3+, Al3+, and Cr3+ on OH defects to improve our understanding of the hydrogen speciation in natural olivine. Low-temperature infrared spectra (−194 °C) are collected on synthetic and natural olivines. These spectra are then interpreted in the light of the theoretical determination of the structural, vibrational, and infrared spectroscopic properties of Fe-related OH defects, using first-principles calculations based on density functional theory. The presence of Fe2+ changes the cationic environment around the fully protonated vacancies in forsterite, leading to a slight modification of their infrared signatures. In particular, the presence of Fe2+ in an octahedral site adjacent to a hydrogarnet-type defect is likely responsible for the additional bands observed at 3599 cm−1 and around 3520–3550 cm−1 in Fe-doped olivines. Results show that the OH bands between 3310 and 3380 cm−1 are associated with the presence of trivalent cations. Specifically, two bands at 3323 and 3358 cm−1, commonly observed in natural olivine, are associated with the substitution of Mg2+ by Cr3+ while two similar bands at 3328 and 3353 cm−1 are associated with the substitution of Mg2+ by Fe3+. The presence of these defects and the “titanoclinohumite” defect in natural olivine clearly underlines the prominent role of trace elements on the hydrogen incorporation in lithospheric olivine.