We construct and examine the prototype of a deep learning‐based ground‐motion model (GMM) that is both fully data driven and nonergodic. We formulate ground‐motion modeling as an image processing task, in which a specific type of neural network, the U‐Net, relates continuous, horizontal maps of earthquake predictive parameters to sparse observations of a ground‐motion intensity measure (IM). The processing of map‐shaped data allows the natural incorporation of absolute earthquake source and observation site coordinates, and is, therefore, well suited to include site‐, source‐, and path‐specific amplification effects in a nonergodic GMM. Data‐driven interpolation of the IM between observation points is an inherent feature of the U‐Net and requires no a priori assumptions. We evaluate our model using both a synthetic dataset and a subset of observations from the KiK‐net strong motion network in the Kanto basin in Japan. We find that the U‐Net model is capable of learning the magnitude–distance scaling, as well as site‐, source‐, and path‐specific amplification effects from a strong motion dataset. The interpolation scheme is evaluated using a fivefold cross validation and is found to provide on average unbiased predictions. The magnitude–distance scaling as well as the site amplification of response spectral acceleration at a period of 1 s obtained for the Kanto basin are comparable to previous regional studies.