The Regional Earthquake Likelihood Models (RELM) working group designed a 5‐year experiment to forecast the number, spatial distribution, and magnitude distribution of subsequent target earthquakes, defined to be those with magnitude ≥4.95 (M 4.95+) in a well‐defined California testing region. Included in the experiment specification were the description of the data source, the methods for data processing, and the proposed evaluation metrics. The RELM experiment began on 1 January 2006 and involved 17 time‐invariant forecasts constructed by seismicity modelers; by the end of the experiment on 1 January 2011, 31 target earthquakes had occurred. We analyze the experiment outcome by applying the proposed consistency tests based on likelihood measures and additional comparison tests based on a measure of information gain. We find that the smoothed seismicity forecast by Helmstetter et al., 2007 based on M 2+ earthquakes since 1981, is the best forecast, regardless of whether aftershocks are included in the analysis. The RELM experiment has helped to clarify ideas about testing that can be applied to more wide‐ranging earthquake forecasting experiments conducted by the Collaboratory for the Study of Earthquake Predictability (CSEP).
Online Material: Figures and tables showing the RELM testing region and collection region definitions, numerical results associated with the RELM experiment, and the uncorrected forecast by Ebel et al. (2007).