It is well known that ground with irregular topographic surfaces causes complicated seismic responses. The complex seismic response is mainly caused by scattering and wave conversions. However, the specific locations of the surface where the scattering mainly occurs and the extent of their effects are not yet clear. In this study, we investigated the excitation process of complicated seismic responses induced by irregular ground surfaces in terms of the contribution of scattered waves. First, the formulation of scattered-wave contribution in a two-dimensional SH-wave field based on the direct boundary element method and the Neumann series expansion of the bem matrix was shown. In the formulation process, it was pointed out that the mathematical expression of the first-order scattered-wave contribution has a form consisting of a wave function and an inclination factor, which was similar to that obtained by the Huygens–Fresnel principle. Next, numerical analyses were conducted for a ground that had a sinusoidal-shaped surface at the center and flat parts at both ends. A comparison of the results showed that the complicated waveforms of the responses were caused by the arrivals of the scattered waves. Finally, the contributions of the first-order scattered waves at the reference points were closely examined based on the mathematical expression; the following conclusions were drawn: (1) The polarity of the first-order scattered waves in the time domain is attributed to the inclination factor, which depends only on the geometrical relationship between the reference point and the source point from which the scattered waves emanate. (2) At the bottom of a valley, the scattered waves generated at its nearby surface are dominant because of the short distance from the source of the scattered waves. These scattered waves appear nearly at the same time of arrival as the incident wave and always reduce the amplitude of the incident wave because of their negative polarity. (3) On the contrary, at the peak of a hill, the scattered waves generated at the nearby surface have positive polarity, and they always enhance the amplitude response.