A highly resolved velocity structure on the mainshock fault of the 2004 Mid-Niigata prefecture earthquake was elucidated using arrival times from aftershocks observed by an extremely dense network of temporary seismic stations. We evaluated the spatial relationship between the velocity structure and distributions of the slip, stress change, and aftershocks. Distributions of static stress drop on the fault were calculated from a kinematic slip model, incorporating the 3D crustal structure. A high-velocity body with low aftershock activity was in proximity to the mainshock hypocenter and extended to the northeast side of the fault. This high-velocity body roughly coincided with an asperity, where the amounts of coseismic slip and static stress drop were larger than those in the surrounding areas. In contrast, a zone of negative stress drop was observed on the shallow periphery of the high-velocity body, where the sediment thickness steeply increased southwestward and the aftershock activity was high. We suggest that the structural heterogeneities of the host rocks surrounding the fault damaged zone have significant potentials to control dynamic rupture processes of the mainshock fault.