A depletion of high‐frequency ground motions on soil sites has been observed in recent large earthquakes and is often attributed to a nonlinear soil response. Here, I show that the reduced amplitudes of high‐frequency horizontal‐to‐vertical spectral ratios (HVSRs) on soil can also be caused by a smooth crustal velocity model with low shear‐wave velocities underneath soil sites. I calculate near‐fault ground motions using both 2D dynamic rupture simulations and point‐source models for both rock and soil sites. The 1D velocity models used in the simulations are derived from empirical relationships between seismic wave velocities and depths in northern California. The simulations for soil sites feature lower shear‐wave velocities and thus larger Poisson’s ratios at shallow depths than those for rock sites. The lower shear‐wave velocities cause slower shallow rupture and smaller shallow slip, but both soil and rock simulations have similar rupture speeds and slip for the rest of the fault. However, the simulated near‐fault ground motions on soil and rock sites have distinct features. Compared to ground motions on rock, horizontal ground acceleration on soil is only amplified at low frequencies, whereas vertical ground acceleration is deamplified for the whole frequency range. Thus, the HVSRs on soil exhibit a depletion of high‐frequency energy. The comparison between smooth and layered velocity models demonstrates that the smoothness of the velocity model plays a critical role in the contrasting behaviors of HVSRs on soil and rock for different rupture styles and velocity profiles. The results reveal the significant role of shallow crustal velocity structure in the generation of high‐frequency ground motions on soil sites.