The main purpose of the site classification or velocity determination at a target site is to obtain or estimate the horizontal site amplification factor (HSAF) at that site during future earthquakes because HSAF would have significant effects on the strong‐motion characteristics. We have been investigating various kinds of methods to delineate the S‐wave velocity structures and the subsequent HSAF, as precisely as possible. After the advent of the diffuse field concept, we have derived a simple formula based on the equipartitioned energy density observed in the layered half‐space for incident body waves. In this study, based on the diffuse field concept, together with the generalized spectral inversion technique (GIT), we propose a method to directly estimate the HSAF of the S‐wave portion from the horizontal‐to‐vertical spectral ratio of earthquakes (eHVSRs). Because the vertical amplification is included in the denominator of eHVSR, it cannot be viewed as HSAF without correction. We used GIT to determine both the HSAF and the vertical site amplification factor (VSAF) simultaneously from strong‐motion data observed by the networks in Japan and then deduced the log‐averaged vertical amplification correction function (VACF) from VSAFs at a total of 1678 sites in which 10 or more earthquakes have been observed. The VACF without a category has a constant amplitude of about 2 in the frequency range from 1 to 15 Hz. By multiplying eHVSR by VACF, we obtained the simulated HSAF. We verified the effectiveness of this correction method using data from observation sites not used in the aforementioned averaging in the frequency range from 0.12 to 15 Hz.