Quantitative Evaluations of Earthquake Early Warning Performance Using “Did You Feel It?” and Post‐Alert Surveys Open Access

The ShakeAlert earthquake early warning (EEW) system for the United States west coast is operated by the U.S. Geological Survey (USGS) in collaboration with university and state partners (Given et al., 2018; Kohler et al., 2020; McGuire et al., 2025). Public EEW alerts via ShakeAlert have been available in California since 2019 and Oregon and Washington since 2021 (McGuire et al., 2025). ShakeAlert ingests real‐time seismic and geodetic data from regional earthquake monitoring networks to detect and characterize earthquakes, and the resulting earthquake source estimates (magnitude and location) alongside estimated ground‐shaking distributions are published as ShakeAlert Messages, which form the basis for the alert regions. Official licensed alerting partners then issue EEW alerts to system users when the ShakeAlert Messages meet predefined magnitude and modified Mercalli intensity (MMI) alert threshold criteria.

ShakeAlert operators do not have direct control over the final alert delivery aspect of the system, which presents challenges and uncertainties for EEW system performance evaluations. We know when ShakeAlert Messages are published, but we neither know how many people received alerts via specific alert delivery mechanisms nor how long it took for the alerts to be delivered. Previous studies suggest that alert delivery latencies can be highly variable and range between one to several seconds (Patel and Allen, 2022; McBride et al., 2023), and thus can significantly impact the actual warning time ShakeAlert can provide. In addition, whereas some specific alert‐delivery operators can potentially track the timing and receipt of their alerts (e.g., Patel and Allen, 2022), those details are not always routinely made publicly available, and alerts sent widely to the general public are not feasibly traceable.

In this work, we show that we can quantify alert performance information, independent of the alerting mechanism, by asking EEW users themselves, using responses to the USGS “Did You Feel It?” (DYFI) survey and supplemental EEW questionnaire (Goltz et al., 2023, 2024). About 15% of DYFI respondents, often in the 1000s for many earthquakes, routinely fill out the EEW supplemental questionnaire. In what follows, we first describe the data for the six earthquakes used in this study, which are recent, widely reported earthquakes with EEW alerts from ShakeAlert. We then compare ShakeAlert’s estimated ground motions against the reported intensities. Next, we discuss the alert efficacy of the warning times, focusing on the range of latencies inferred from actual alert recipients. Finally, we discuss how these data might best inform how to improve communications about what we know about alerting in general.

We focus our analysis on the six California earthquakes with the largest DYFI responses since the release of the EEW questionnaire (Goltz et al., 2024) and that had ShakeAlert source estimates meeting the criteria for issuing public alerts to mobile phones. The most common public alert thresholds are: M 5.0+ to MMI IV+ locations, used for alerts delivered via the Wireless Emergency Alert (WEA) system (McBride et al., 2023); and M 4.5+ to MMI III+ locations for alerts delivered via dedicated EEW applications (apps) (such as MyShake; Patel and Allen, 2022) and via Android operating systems (Allen and Stogaitis, 2022).

Details of the ShakeAlert performances for these earthquakes are as follows (and summarized in Table S1):

20 August 2023 M 5.1 Ojai earthquake: ShakeAlert detected this earthquake at 6.7 s after origin time with an initial (and maximum) magnitude estimate of M 6.0, which caused the MMI IV+ and MMI III+ alert regions to be initially restricted by ShakeAlert’s “alert pause” feature. The alert pause is a configuration where, for the first 5 s from the time of the first alert, alert regions are restricted to radii of ≤100 km (McGuire et al., 2025). After 5 s, ShakeAlert publishes unrestricted alert regions for its latest source estimate, and all subsequent updates are unrestricted. This configuration was implemented to reduce the potential impact of over‐alerting caused by significant magnitude overestimation, which is more likely to occur during the initial alerts, as these use observations from as few as four stations (Lux et al., 2024; McGuire et al., 2025). For the Ojai earthquake, the peak magnitude estimate once the alert pause ended was M 5.7, resulting in significant expansion of the MMI III+ alert region. The highly populated Los Angeles area was included in the MMI IV+ alert region used for WEA alerts, and the MMI III+ alert region used in other mobile‐phone‐based alerts extended into San Diego.

7 August 2024 M 5.2 Lamont earthquake: ShakeAlert’s initial alert was at 7.7 s after origin time with a magnitude estimate of M 6.0. This was the overall maximum magnitude estimate, but alert regions were restricted by the alert pause. The maximum magnitude estimate after the alert pause ended was M 5.7. A WEA alert was issued for this earthquake, with the MMI IV+ alert region primarily to the north of the Los Angeles area, though all of Los Angeles was included inside the MMI III+ alert region.

12 August 2024 M 4.4 Highland Park earthquake: ShakeAlert had an initial magnitude estimate of M 4.4 for this earthquake at 4.2 s after origin time, but later updates reached a maximum magnitude estimate of M 4.7. This met the criteria for issuing public alerts via EEW mobile phone apps and Android operating systems to locations estimated to experience MMI III+ shaking, which included the majority of the Los Angeles area. This is the only earthquake in our dataset that did not meet the WEA alert threshold.

12 September 2024 M 4.7 Malibu earthquake: ShakeAlert detected this earthquake at 5.2 s after origin time with an initial magnitude estimate of M 5.0 and a later update reaching a peak magnitude estimate of M 5.1, resulting in a WEA alert being issued to MMI IV+ estimated locations as well as mobile‐phone‐based alerts issued to MMI III+ estimated locations in the Los Angeles area. The alert regions were small enough that they were not truncated by the alert pause.

5 December 2024 M 7.0 offshore Cape Mendocino earthquake: ShakeAlert had an initial alert of M 5.6 at 15.1 s after origin time. The location of the initial alert was centered offshore, but later alert updates had a ShakeAlert location estimate onshore, pulled toward the network stations. Although the alert regions during the beginning of the alert sequence were restricted by the alert pause, the alerts associated with ShakeAlert’s maximum magnitude estimate of M 6.9 occurred after the alert pause and were large enough to be centered around a rapid finite‐fault estimate (Böse et al., 2023). The maximum extent of the MMI III+ alert region included northern California and the southwestern portion of Oregon.

14 April 2025 M 5.2 Julian earthquake: ShakeAlert’s initial alert was at 4.5 s after origin time with a magnitude estimate of M 4.8, but later updates (after the alert pause) reached a magnitude estimate of M 6.7 due to high peak ground accelerations at stations within 50 km of the epicenter. The MMI IV+ region used for alerts via WEA included most of southern California and all of the San Diego and Los Angeles areas.

ShakeAlert aims to produce ground‐motion estimates corresponding to median observations (Given et al., 2018; McGuire et al., 2025). This approach enables the variety of ShakeAlert users to take different alert threshold strategies based on their needs, such as their shaking thresholds of interest for receiving alerts and cost‐benefit trade‐offs associated with overalerting (e.g., Minson et al., 2019, 2022; Saunders et al., 2022). Despite this emphasis on matching median ground‐motion observations, when it comes to evaluating ShakeAlert’s performance, most public‐facing depictions focus on the relative accuracy of its source estimates compared to the authoritative catalog source values and assume that source characterization accuracy relates directly to alert region accuracy.

For the earthquakes in our dataset, all but one had ShakeAlert magnitude estimates higher than the catalog magnitudes, which, using source‐based comparisons only, implies that ShakeAlert overestimated the ground motions for these earthquakes. However, this relationship between magnitude accuracy and alert region accuracy may not be as direct, in part due to ground‐motion variabilities that are difficult to model in an EEW context (Minson et al., 2022). In addition, the alert pause feature complicates how source‐based performance comparisons relate to alert region accuracy. If ShakeAlert’s maximum magnitude estimate occurs during the alert pause and later unrestricted alert updates use lower magnitude estimates, the maximum extent of the alert regions may not be associated with the peak magnitude alert. Similarly, the maximum extents of specific alert thresholds may no longer be associated with the same ShakeAlert Message update if one is restricted during the alert pause and the other is not.

We instead favor performance evaluations that directly compare the predicted ground motions from ShakeAlert with observed ground motions, specifically DYFI reports. DYFI reports provide independent macroseismic intensities based on observed impacts and human experiences that define the intensity at any location (Dengler and Dewey, 1998; Wald et al., 2011). Although comparisons can be made between estimated and observed seismic station data, such as peak ground accelerations and velocities, this only corresponds to an intermediate step in the ground‐motion estimation procedure. The full alert region calculation requires conversion from these instrumental ground‐motion metrics to MMI, as ShakeAlert’s alert regions and public alert thresholds are based on MMI. This calculation is nontrivial; the choice of conversion equation significantly impacts alert region size, and the conversion procedure used in ShakeAlert cannot be used for individual instrumental observations (Saunders et al., 2024). Thus, our approach benefits from (1) not needing to convert from recorded ground‐motion metrics to intensity and to account for different MMI conversion equations, and (2) comparing ShakeAlert intensity predictions, which dictate alert domains, directly with human observations.

Figure 1 shows MMI with distance curves predicted by ShakeAlert compared against the observed intensities from DYFI reports for the earthquakes in our analysis. We show the maximum‐predicted MMI from the entire alert sequence, incorporating the effects of the alert pause, alongside what ShakeAlert would have predicted using the authoritative catalog magnitude as input. These comparisons confirm that magnitude estimation accuracy is not necessarily indicative of ground‐motion prediction accuracy. The M 7.0 offshore Cape Mendocino earthquake had very close ShakeAlert and catalog magnitudes, yet shaking is overpredicted at distances >40 km. Some overprediction could be due to ShakeAlert’s onshore source location estimate 39 km to the east of the catalog epicenter (Table S1); however, the ShakeAlert ground‐motion models have been noted to overestimate intensities for higher magnitudes (Saunders et al., 2024).

ShakeAlert significantly overestimated the magnitudes of the M 5.2 Julian, M 5.2 Lamont, and M 5.1 Ojai earthquakes, resulting in ground‐motion predictions >1 standard deviation higher than median observations (Fig. 1b–d). The predictions generally correspond with the maximum‐reported shaking from DYFI, though the alert pause reduced some overprediction at >100 km distances for the Lamont and Ojai earthquakes. However, accurate magnitude estimates would have underpredicted shaking relative to median observations in many locations, particularly for the Julian earthquake and for near‐source locations for the Ojai earthquake (Fig. 1b,d).

For the M 4.7 Malibu and M 4.4 Highland Park earthquakes, ShakeAlert magnitude overestimation did not result in ground‐motion overprediction (Fig. 1e,f). In fact, the predicted ground motions for the Malibu earthquake matched well with median DYFI observations, whereas shaking for the Highland Park earthquake was still underpredicted. Accurate magnitude estimates would have underpredicted shaking by >1 standard deviation of median observations for both earthquakes. In addition, this would have prevented EEW alerts from being issued for the Highland Park earthquake, which caused minor damage (Gonzales et al., 2024) and had many reports of MMI VI, a primary target for EEW alerts in the ShakeAlert system (McGuire et al., 2025).

We now examine the responses to the DYFI EEW questionnaire (Table S2). Our objectives are to evaluate the EEW performance in terms of who received an alert and, if so, how much warning time they received, and then compare these results against what we can calculate using the ShakeAlert Message content and publication times. We specifically focus on the responses to the first three questions (Goltz et al., 2023):

• Did you or someone near you receive an earthquake alert?

• Did you receive an alert message before, during, or after you felt shaking?

• If you received an alert before you experienced shaking, how many seconds did the alert arrive before you felt shaking?

We defer examination of the other questions concerning how respondents felt about and reacted to the alert messages to later studies (though for analysis of the M 5.1 Ojai earthquake, please refer to Goltz et al., 2024).

A key challenge is that the specific alert delivery mechanism for any given response is unknown. Although people can supply this information as part of an open‐ended response at the end of the survey, questions about U.S.‐specific alert delivery mechanisms are not included because the EEW questionnaire is designed to apply to EEW systems generally (Goltz et al., 2023). However, we can narrow down what types of alerts could have been issued to a given location using known public alert thresholds. For example, an earthquake estimated by ShakeAlert to be M ≥ 5.0 will have WEA alerts in locations with MMI IV+ estimates, whereas other mobile‐phone‐based alerts can extend to MMI III+ locations. With the overlap in alert thresholds, many people may receive multiple alerts, for example, through both WEA and MyShake. For these cases, we assume that the reported warning time is associated with the earliest alert they received and make no attempt to untangle which alert delivery mechanism was the fastest.

We investigate the distribution of DYFI EEW questionnaire responses for the earthquakes in our study in terms of map comparisons (Fig. 2) and epicentral distance comparisons (Fig. 3). We include the maximum‐predicted MMI extents across the alert sequence for the main public alert thresholds used in ShakeAlert. The map comparisons also include the observed MMI from ShakeMap (Wald et al., 2022). In addition, we compare the reported warning time distribution relative to maximum‐expected warning times calculated using the time of the first M 4.5+ ShakeAlert Message publication (the lowest magnitude threshold for public alerts to mobile phones) and the expected S‐wave arrival time (using a constant velocity of 3 km/s; Minson et al., 2020; Cochran et al., 2022). We incorporate a 5 s delay in warning times for >100 km locations due to the alert pause; however, we do not include any assumed alert delivery latencies.

Examination of alert receipt reports from the DYFI EEW questionnaire confirms that EEW alerts are issued within the expected public alert regions from ShakeAlert (i.e., within MMI III+ locations in the states where ShakeAlert operates). From the warning time comparisons, we observe that nearly all reports within a given perceived warning time category occur at distances at or beyond where that category is expected to begin according to the maximum‐estimated warning time from ShakeAlert. The lack of longer‐than‐expected warning time reports further indicates that the reported warning times are reasonably accurate.

However, there are many reports of not receiving an alert within the ShakeAlert MMI III+ regions as well as significantly shorter warning times and late alerts in locations where longer warning times (>10 s) are expected. Some of this behavior appears to be mitigated where WEA alerts were issued. For all earthquakes, in locations with a WEA alert, 4.7% of respondents reported not receiving an alert, whereas in locations where a WEA alert was not issued but other alerts to mobile phones would have been issued, 39.6% of respondents reported not receiving an alert (Fig. S7). These differences are not unexpected; WEA alerts are issued to all mobile phones, whereas the other mobile‐phone‐based ShakeAlert delivery mechanisms require a person to have a specific model of smartphone or to download a specific app.

WEA alert issuance does not appear to increase warning times. Of all respondents who received alerts, 49.2% reported late alerts (alerted during or after shaking), where only 5.5% of the reported late alerts are located inside the estimated initial late alert zones. For respondents who could have received WEA alerts, 51.0% reported late alerts, with 5.8% in the initial late alert zones. For alerted respondents outside of WEA alert regions, 53.4% reported late alerts, with 4.5% inside the initial late alert zones. Controlled tests of WEA delivery latencies indicate that alert deliveries may be fairly slow, with an average of ∼8 s (McBride et al., 2023). Analyses of EEW app alert delivery times indicate that their latencies are variable but much shorter on average, and that latencies may increase when alerting to larger populations simultaneously (Patel and Allen, 2022). The observed consistency in the late alert distribution across alerting types indicates that alert delivery latencies may be substantial, including for app‐based alerts.

The DYFI EEW questionnaire responses confirm that people are indeed receiving actionable EEW alerts for earthquakes in the U.S. Direct comparisons to reported intensities in DYFI (Figs. 1, 4) and in ShakeMap (Fig. 2) emphasize how the observed variability in shaking intensity is not well captured by the necessarily simple ground‐motion estimation procedures used in ShakeAlert, as there are many people who report experiencing different shaking intensities compared to the median predictions from ShakeAlert. These types of comparisons could be important visualization tools in alert performance communications because they contextualize the alert information with the shaking that people experienced.

The distributions of DYFI questionnaire responses confirm expected alert performance behavior, where people who are closer to the earthquake source and who experience higher intensities receive shorter warning times, and people who report longer warning times are farther away and generally experience lower intensities (Figs. 3, 4). There are substantially fewer reports of >10 s warning compared to shorter warning times, which is unexpected given the wide areas where long warnings are expected from the ShakeAlert Message publication times, including several earthquakes where >30 s warning is possible in highly populated regions (Figs. 2, 3). This apparent discrepancy was also observed for the Earthquake Network app during the 2023 M 7.8 Türkiye earthquake, where alerts were calculated to provide >30 s warning before the arrival of significant shaking in many populated areas (Finazzi et al., 2024). From surveyed app users after the earthquake, only 26% indicated they received an alert before experiencing shaking, and most of these reported warning times of <15 s (Bossu et al., 2025). It is possible that the shorter reported warning times could reflect additional time required to understand the alert information after the alert begins sounding on their phones. Because our DYFI dataset represents only a small portion of the potentially millions of people who were issued alerts for the earthquakes in our study, asking official ShakeAlert delivery partners to link alert recipients to the DYFI survey and EEW questionnaire may increase reporting for future earthquakes and help confirm whether these shorter‐than‐expected warning times are indeed prevalent.

In terms of discussing public EEW performance, the distribution of positive warning time reports indicates that calculating the initial late alert zone using the S‐wave extent at the time of the first alert is a reasonable assumption (e.g., Wald, 2020; Cochran et al., 2022). Whereas other EEW analyses may focus on precise warning time calculations relative to the measured exceedance time of a given target shaking threshold (e.g., Meier et al., 2020; McGuire et al., 2025), the observed variability in reported warning times indicates that showing precise estimates would be inaccurate to many people’s alert experiences. The choice of target shaking threshold would also be complicated, as public preferences for EEW alerts in the U.S. range between damaging to felt shaking thresholds (Bostrom et al., 2022). Indeed, the intensity distribution of people who reported that they did not receive an alert but expected one includes respondents who experienced weak shaking (Fig. 4). The S‐wave arrival is also consistent with the phrasing in the questionnaire, which asks about the timing of the alert relative to when the respondent started feeling shaking. Although it is likely that for larger earthquakes the near‐source P‐wave amplitudes may be large enough to be widely felt (e.g., Minson et al., 2018, 2020; Thompson et al., 2024), the largest earthquakes in our dataset do not contain reported alert information at close enough source distances to test this.

Although we focus on six California earthquakes, many people in other locations, including those outside the United States, routinely submit DYFI reports (for a map of global responses, see Data and Resources). Given that the DYFI EEW questionnaire is agnostic to the source of the EEW alert, other DYFI and similar macroseismic systems worldwide could adopt it for wider data collection or incorporate these questions into their post‐alert surveys. Such responses can be useful for ShakeAlert and other EEW systems worldwide to improve understanding about operational EEW alert performance and public perceptions of EEW alerts.

We analyzed the EEW performance for six widely felt California earthquakes that had public alerts from the U.S. ShakeAlert EEW system using the shaking intensities, alert receipt information, and available warning times as reported in the USGS DYFI survey and supplemental EEW questionnaire. We found that source‐parameter‐based alert assessments cannot provide a full picture of EEW performance. Direct comparisons to observed shaking (like independent DYFI intensities) provide better insight into whether alert regions were over‐ or underpredicted. Comparisons of ShakeAlert Message publication times to reported warning times from DYFI indicate that estimating maximum‐expected warning times relative to the S‐wave arrival is appropriate for public EEW performance discussions, and that communicating precise warning time estimates may be inaccurate given the reported warning time and alert receipt variabilities. The percentage of people who report receiving an alert increases in locations where WEA alerts are issued. Reported alert timeliness information shows later‐than‐expected alerts in all alert regions, indicating that alert delivery latencies may be highly variable and significant for all alert delivery mechanisms. Our results can be leveraged alongside existing earthquake information products to help improve the public‐facing alert summary information for the ShakeAlert system as well as for other EEW systems in the future.

Earthquake source information, ShakeMaps, and aggregated “Did You Feel It?” (DYFI) intensities were obtained from the Advanced National Seismic System (ANSS) Comprehensive Catalog (ComCat; U.S. Geological Survey [USGS], 2017); earthquake.usgs.gov, last accessed April 2025). ShakeAlert information for earthquakes with public alerts is available online via ComCat on each event’s webpage. The U.S. Geological Survey (USGS) permanently archives individual DYFI survey and DYFI earthquake early warning (EEW) questionnaire responses; anonymized individual responses are available by request at [email protected]. A map of global responses is available at https://earthquake.usgs.gov/data/dyfi/ (last accessed June 2025). Additional tables and figures are included in the supplemental material to this article.

The authors acknowledge that there are no conflicts of interest recorded.

J. K. Saunders was supported by U.S. Geological Survey (USGS) ShakeAlert Cooperative Agreement G24AC00477‐00. The authors thank Vince Quitoriano (USGS contractor) for facilitating access to the “Did You Feel It?” (DYFI) earthquake early warning (EEW) supplemental questionnaire responses. The authors also thank Sarah Minson, Rémy Bossu, an anonymous reviewer, the editors at The Seismic Record, and the ShakeAlert community for helpful comments and discussion that improved this work.

Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.