Since the early days of modern seismology, toppled artifacts such as tombstones and single columns have been used in the aftermath of earthquakes to deduce parameters of site-specific ground motions. The artifacts were generally treated as rigid bodies. Later, the theory of rigid block movements was also applied to precariously balanced rocks toppled by earthquakes. While the movements of a single rocking block can be described analytically, slide-rocking movements, bouncing, and multiple block systems require a numerical approach. We use multiple rigid block models with viscoelastic coupling forces in combination with full 3D ground motions (measured and synthetic) to analyze the dynamic response of building elements, relevant for archaeoseismological studies. First, the numeric modeling results are verified by comparison with analytically determined rocking motions of a single rectangular block. Stiffness and damping parameters of the coupling forces are adjusted to results from analog experiments with a rocking marble block. A model of a monolithic column and one consisting of seven drums is used to test the influence of the geometry and friction on the toppling behavior. The main question addressed in this study is whether toppled columns give a clear indication of the back azimuth toward the earthquake source. Input motion from 29 strong-motion records indicates little correlation between downfall directions and back azimuth. Clearly directed horizontal ground movements tend to topple the columns in the transverse direction. More complex ground motions result in quasi-random downfall directions. The friction coefficients have a minor influence on the downfall directions. Synthetic ground motions for two earthquakes with different source mechanism show toppling directions toward and away from the source as well as in the transverse bearing. However, it is not straightforward to deduce a reliable source location from the inversion of the toppling directions.