Empirical transfer functions (ETFs) between seismic records observed at the surface and depth represent a powerful tool to estimate site effects for earthquake hazard analysis. However, conventional modeling of site amplification, with assumptions of horizontally polarized shear waves propagating vertically through 1D layered homogeneous media, often poorly predicts the ETFs, particularly, in which large lateral variations of velocity are present. Here, we test whether more accurate site effects can be obtained from theoretical transfer functions (TTFs) extracted from physics‐based simulations that naturally incorporate the complex material properties. We select two well‐documented downhole sites (the KiK‐net site TKCH05 in Japan and the Garner Valley site, Garner Valley Downhole Array, in southern California) for our study. The 3D subsurface geometry at the two sites is estimated by means of the surface topography near the sites and information from the shear‐wave profiles obtained from borehole logs. By comparing the TTFs to ETFs at the selected sites, we show how simulations using the calibrated 3D models can significantly improve site amplification estimates as compared to 1D model predictions. The primary reason for this improvement in 3D models is redirection of scattering from vertically propagating to more realistic obliquely propagating waves, which alleviates artificial amplification at nodes in the vertical‐incidence response of corresponding 1D approximations, resulting in improvement of site effect estimation. The results demonstrate the importance of reliable calibration of subsurface structure and material properties in site response studies.