Following the 2011 magnitude 9.1 Tohoku earthquake, Geller (2011) argued that “all of Japan is at risk from earthquakes, and the present state of seismological science does not allow us to reliably differentiate the risk level in particular geographic areas,” so a map showing uniform hazard would be preferable to the existing map. We explore this by comparing how well a 510‐yr‐long record of earthquake shaking in Japan is described by the Japanese national‐hazard (JNH) maps, uniform maps, and randomized maps. Surprisingly, as measured by the metric implicit in the JNH maps (i.e., a metric that requires only a specific fraction of the sites during the chosen time interval to exceed the predicted ground motion), both uniform and randomized maps do better compared with the actual maps. However, using the squared misfit between maximum observed shaking and the predicted shaking as a metric, the JNH maps do better compared with the uniform or randomized maps. These results indicate that (1) the JNH maps are not performing as well as expected, (2) identification of the factors controlling map performance is complicated, and (3) learning more about how maps perform and why would be valuable in making more effective policy.