Raman spectra of quartz inclusions in garnet hosts of low-pressure/temperature metamorphic rocks from the Yanai district in the Ryoke belt (around 0.1–0.3 GPa/500–600 °C), Southwest Japan, exhibit frequency (peak position) shifts toward lower wavenumbers as compared to those of a quartz standard measured at ambient conditions. The observed negative frequency shifts indicate that tensile normal stress is exerted on the quartz–garnet boundary and therefore, quartz inclusions are subjected to negative residual pressure. Elastic modeling that assumed the constant elastic properties of minerals cannot explain this negative residual pressure. This study estimated the residual pressure based on a new scheme of elastic modeling with equation of state (EOS) of quartz and garnet, which takes into account the pressure- and temperature-dependency of compressibility and expansivity. The calculated residual pressure was converted into frequency shifts of quartz Raman spectrum based on the experimentally determined relation. The results showed that the quartz inclusions in garnets retain residual pressure of about −0.3 GPa, and logically reproduced the observed frequency shifts in the direction of lower wavenumbers. The new elastic modeling also simulates positive frequency shifts retained by quartz inclusions in garnets of high-pressure metamorphic rocks from the Sambagawa metamorphic belt in Southwest Japan, and from the Motagua fault zone in Guatemala. The degree and direction of Raman frequency shifts of quartz inclusion in garnet depend on metamorphic conditions when the quartz was included in the host garnet. Conversely, the metamorphic conditions prevailing when a set of a quartz inclusion and garnet host was recrystallized can be inferred from Raman frequency shifts of quartz inclusion in garnet. The proposed Raman spectroscopic analysis should be a powerful and useful tool to decipher information at earlier stage of garnet growth even in samples of highly recrystallized matrix phases during exhumation and retrograde stages.