In this study we apply a nonparametric spectral inversion scheme to a data set of accelerograms recorded by the K-NET and KiK-net networks in Japan in order to derive attenuation characteristics, source spectra, and site response. For this purpose, we use a total of more than 67,000 S-wave records from 2178 earthquakes (MJMA 2.7–8) obtained at 1555 stations at the Earth’s surface and more than 29,000 records from 1826 events recorded at 637 borehole stations at depths of 100 to 3000 m. Attenuation characteristics are investigated in five separate regions, showing that crustal Q depicts lower values in central compared to southern Japan, and a significant frequency dependence is observed in every region. The source spectra follow the ω2 model with higher stress drops for subcrustal earthquakes as compared with crustal ones. While strong amplification effects dominate the site contributions for the surface sensors, those for the borehole sensors are characterized by smaller variability. Nevertheless, consistent with observations from deconvolution of borehole/surface recording pairs, downgoing wave effects are visible in the site contributions for many borehole stations. Finally, the site amplification functions obtained at the surface are compared with surface-to-borehole (S/B) and horizontal-to-vertical (H/V) spectral ratios, showing that the S/B ratios generally provide better estimates of the horizontal amplification than the H/V ratios due to amplification of the vertical component of ground motion.