We analyzed temporal changes in site response associated with the strong ground motion of the 2004 Mw 6.6 Mid-Niigata earthquake sequence in Japan. The seismic data were recorded at a site with accelerometers at the surface and a 100-m-deep borehole. We computed the empirical surface-to-borehole spectral ratios and used them to track temporal changes in the top 100 m of the crust. We observed that the peak spectral ratio decreases by 40%–60% and the peak frequency drops by 30%–70% immediately after large earthquakes. The coseismic changes are followed by apparent recoveries, with the time scale ranging from several tens to more than 100 sec. The coseismic peak frequency drop, peak spectral ratio drop, and the postseismic recovery time roughly scale with the input ground motions when the peak ground velocity is larger than ∼5 cm/sec (or the peak ground acceleration is larger than ∼100 Gal). Our results suggest that at a given site the input ground motion plays an important role in controlling both the coseismic change and the postseismic recovery in site response.