This article presents the results of a shaker test of the Millikan Library in Pasadena, California, using sensors inside smartphones to demonstrate their potential usage as a way to monitor health states of buildings. This approach to structural health monitoring could allow many more commercial and residential buildings to be monitored because it removes the cost prohibitive nature of traditional seismic arrays and the complexity of deploying the instruments. Recordings from the smartphones during the shaking show high correlation with those from a reference sensor in the building, illustrating that the phones can capture the shaking even when not fully coupled to the floor. The fundamental translational frequencies for the east–west and north–south directions and the torsional frequencies of the building can be extracted from single phone recordings. As we compare the displacement derived from the phone recording by double integration to that from the reference sensor, both phase and amplitude match well. Signal‐to‐noise ratio is improved further by stacking records from multiple phones. These test results demonstrate the ability to extract the fundamental translational and torsional frequencies, and absolute displacements from upper levels of buildings shaken by small local earthquakes. This work builds on the ongoing MyShake project—a global smartphone seismic network.