Damping modification factors (DMFs) are important for estimating the response spectrum for the design of structures with different damping ratios. This study investigated the effects of earthquake source parameters (magnitude, source depth, and source categories), source distance (the closest distance from a site to a fault plane for large earthquakes and hypocentral distance for the other events), and site conditions on DMFs for the displacement and acceleration response spectra of the horizontal components of the records from Japan. This study used a total of 14,713 strong‐motion records from the KiK‐net and K‐NET to compare the DMFs from three earthquake category groups, namely shallow crustal and upper mantle, subduction interface, and subduction slab earthquakes. Statistical tests were carried out to determine whether the DMFs from these three types of earthquakes differ significantly from each other. The test results show that, between each pair of the three types of earthquakes, the DMFs for both displacement and acceleration response spectra differ significantly in terms of statistical tests and practically for engineering applications at many spectral periods, with the largest difference over 40%. The effects of earthquake category and site conditions for acceleration spectrum are similar to those of the displacement spectrum at short periods up to about 0.3 s but are much larger than those of the displacement spectrum at long spectral periods. The effects of magnitude and earthquake depth are also significant. Therefore, separate DMF models for the response spectrum of the horizontal component should be derived for each type of earthquake and should account for the effects of earthquake source and path parameters and site conditions.