We present the spectral element method to simulate elastic-wave propagation in realistic geological structures involving complieated free-surface topography and material interfaces for two- and three-dimensional geometries. The spectral element method introduced here is a high-order variational method for the spatial approximation of elastic-wave equations. The mass matrix is diagonal by construction in this method, which drastically reduces the computational cost and allows an efficient parallel implementation. Absorbing boundary conditions are introduced in variational form to simulate unbounded physical domains. The time discretization is based on an energy-momentum conserving scheme that can be put into a classical explicit-implicit predictor/multi-corrector format. Long-term energy conservation and stability properties are illustrated as well as the efficiency of the absorbing conditions. The associated Courant condition behaves as ΔtC < O (nel−1/ndN−2), with nel the number of elements, nd the spatial dimension, and N the polynomial order. In practice, a spatial sampling of approximately 5 points per wavelength is found to be very accurate when working with a polynomial degree of N = 8. The accuracy of the method is shown by comparing the spectral element solution to analytical solutions of the classical two-dimensional (2D) problems of Lamb and Garvin. The flexibility of the method is then illustrated by studying more realistic 2D models involving realistic geometries and complex free-boundary conditions. Very accurate modeling of Rayleigh-wave propagation, surface diffraction, and Rayleigh-to-body-wave mode conversion associated with the free-surface curvature are obtained at low computational cost. The method is shown to provide an efficient tool to study the diffraction of elastic waves by three-dimensional (3D) surface topographies and the associated local effects on strong ground motion. Complex amplification patterns, both in space and time, are shown to occur even for a gentle hill topography. Extension to a heterogeneous hill structure is considered. The efficient implementation on parallel distributed memory architectures will allow to perform real-time visualization and interactive physical investigations of 3D amplification phenomena for seismic risk assessment.