In oil and gas exploration, technical teams build subsurface assessments from many data sources. From these assessments, teams extract values that quantify a prospect’s resource uncertainty and the probability of success. These values feed economic evaluations and become the building blocks of an exploration portfolio. Predictable portfolio management depends upon the reliability and accuracy of these prospect quantifications. Assurance teams can provide technical teams with objective, independent, and consistent guidance for these assessments to enhance predictive accuracy. In this paper, the authors summarize our 40 yr of experience characterizing exploration opportunities, participating in or leading company assurance teams, and collaborating with industry assurance practitioners to share key attributes of an effective assurance process. Assurance teams are most effective when they (1) serve as independent groups engaging with technical teams; (2) have a defined yet flexible process; (3) have experienced team members with good interpersonal, listening, and writing skills; (4) can assess opportunities multiple times during the life cycle of a prospect; and (5) conduct and report on postwell analysis to drive positive change. The desired outcome of effective exploration assurance is management being confident that the company technical evaluations are consistent and reliable. This provides a sound basis for managing a portfolio and making informed decisions.
THE PERSISTENT RESOURCE ESTIMATION CHALLENGE
The extent of the resource overestimation problem became apparent at the 1995 AAPG Hedberg Conference dedicated to petroleum risk management (MacKay, 1996). Otis and Schneidermann (1997) provided an early glimpse of Chevron’s poor predictive accuracy of resource estimation in advance of their commitment to an assurance initiative. Harper (2000) described similar resource overestimation problems at BP, particularly as it related to their exploration of deep-water prospects, which were investigated with their most advanced predrill technologies. More recent publications (Ward and Whitaker, 2016; Milkov, 2017) and presentations (Bagley and Bond, 2018; Søiland, 2019) suggest that accurate resource estimation remains a challenge.
Figure 1 illustrates the problem, indicating the overconfidence bias at companies operating in the Norwegian North Sea (Søiland, 2019). Predrill resource estimates in license applications largely overestimated resource potential, predominantly falling in the shaded triangle below the central diagonal perfect prediction line. Many predrill estimates were overestimated by a factor of 10 and some by nearly a factor of 100.
This problem afflicts explorers around the world. The data from Figure 1 are shown as a black line, recast in percentile histogram format in Figure 2. The y axis records the percentage of discoveries that fell in a particular percentile interval from that prospect’s predrill estimated ultimate recoverable resource (EUR) distribution. Each gray line represents different companies’ predictive performance exploring in various basins worldwide during the last three decades. The high percentage of discoveries falling beneath the 80th percentile (P80) of the predrill EUR distribution indicate consistent overestimation of prospect resources. By definition, only 20% of the outcomes should fall below the P80. (This paper uses the “greater than” convention, where P20 > P80). Ideally, in an unbiased estimating system, 20% of the discoveries would fall in each quintile.
Many reasons exist for the persistent overestimation. Capen (1976) first addressed the universal tendency to understate the amount of uncertainty in prospect parameter estimates. Rose and Citron (2000) described the unfailing persistence and eternal optimism associated with the “prospector myth.” Kahneman (2011) summarized overconfidence and other categories of bias that he and his colleague Amos Tversky researched for five decades. They described the difficulty of eliminating bias, a point Rose (2017) corroborated for the oil and gas industry.
THE CASE FOR ASSURANCE IN PORTFOLIO MANAGEMENT
Assurance describes the process of providing independent reality and perspective checks on resource estimation parameters and chance assessment. This helps in reducing bias and providing justified confidence to those who make business decisions. Larger exploration and production (E&P) companies likely started rigorous assurance procedures in the 1980s (Sluijk and Parker, 1986). Adoption of assurance processes became more widespread in the early 1990s, partly in response to failed growth efforts that destroyed shareholder value (Hart, 1988; Rose, 2001, 2012).
Otis and Schneidermann (1997) documented the positive impact of assurance on Chevron’s predictive performance. Although their paper outlined the process for predrill volume and chance assessment, the greatest impact on predictability came from coupling the assessment process with rigorous assurance (R. Otis, 2003, personal communication). Twenty years later, Rudolph and Goulding (2017) reported on the calibration of their compiled chance of economic success forecasts to the actual economic success rate for 553 ExxonMobil conventional prospects drilled between 1994 and 2015. Although they do not mention the word “assurance,” they do refer to (1) prospect assessments that were further evaluated in a structured peer review by senior advisors; (2) longstanding adherence to worldwide assessment standards; and (3) application of a unified upstream risk model (Sykes et al., 2011) to facilitate consistency. Van Mastrigt and Quinn (2021) highlighted, among other factors, that careful assurance of exploration workflows frequently led to outcomes that were consistent with predictions. In addition to these publications, several other presentations (e.g., McMaster, 1997; McMaster and Carragher, 2003; Moeller, 2015; Boyd, 2019) documented improved predictive and business performance through assurance.
Although publications from two of the largest international explorers either allude to (Otis and Schneidermann, 1997) or infer (Rudolph and Goulding, 2017) the utility of assurance, no publication has specifically addressed the design and implementation of an assurance process as part of its portfolio management system. Before we address this, the following section details some precursory efforts that can facilitate a practical approach to assurance.
AN INTEGRATED APPROACH TO BEST PRACTICE EVALUATIONS
The only true alternative to assurance would be a complete lack of preinvestment oversight, which would not be tolerated in any industry. McMaster (2008) noted that if prospect assessment methodology were standardized and consistently applied across the company without bias, then an assurance team would not be needed. He concluded, however, that until this happens, an assurance team must be considered an exploration best practice.
Various design and implementation elements will differ across various organizations; however, there are some fundamental practices that should be consistent in all organizations.
We note some criticisms of assurance. Examples include (1) viewing assurance as a distraction that creates delays in project completion; (2) biased team members’ stifled creativity; or (3) that dedicated, experienced personnel could have been used elsewhere in the business. Therefore, acknowledging and understanding any criticisms of assurance, if warranted, will better serve successful assurance. While these observations are important, they likely do not reflect a problem with the concept of assurance in portfolio management. Rather, they tend to be associated with a flaw in the assurance design or implementation. Before addressing these, we note that it helps to understand some prerequisites that have facilitated assurance, and how to extract the most value from assurance.
Prerequisites to Assurance
Kahneman et al. (2021) found that good judgment comes from people who are not only experienced and intelligent but also willing to learn. Accordingly, there are several prerequisites that may be considered an alternative approach, but in fact complement assurance. Technical staff training should include best practice technical evaluation and uncertainty characterization that covers bias awareness and mitigation. Prospect assessment guidelines promote consistency and methods that have best served the company. Subject matter experts should be consulted as needed. Appropriate incentives can reinforce and reward estimation accuracy as well as better business performance (McMaster and Carragher, 2003). For more on quality control concepts, see Milakovich (1995).
In describing supportive pillars to successful exploration, Latham and Parkinson (2013) noted better managed companies encouraged
the need to characterize the uncertainty by simply stating they do not want surprises;
access to data and tools to provide context for reality checking a prospect;
learning from failures by making postdrill reviews part of the assessment process; and
pairing exploration innovators with others who help them better navigate company assessment processes.
Assurance as a Means for Integration
The quality control items described above provide the first line of defense against potential inconsistencies in subsurface analysis. Assurance at the very least provides the last line of defense before the investment decision. When integrated, quality control and assurance are more effective than either used alone.
Across an organization, assurance reviews provide additional perspectives on prospect assessment, share best practices, identify weaknesses in evaluations, provide alternatives, and foster consistency. Assurance teams are in a unique position to navigate two interfaces in their companies; they must interact both with staff to assist with opportunity characterization and management to aid in opportunity comparison (Figure 3). To perform this function, assurance needs to
define common terminologies and use standardized software;
establish and maintain a consistent, systematic, and calibrated assessment process;
implement that process via predrill reviews and provide recommendations as needed to craft unbiased estimates;
bring about calibration using postdrill evaluations, comparing those evaluations with predrill estimates and recommend improvements for positive change; and
provide guidance to management on the technical robustness of the prospect inventory.
Because there are so few peer-reviewed publications dedicated to the role of assurance in the E&P business, and given the persistence of inaccurate estimates, this paper recommends both design and implementation aspects of assurance and practices to avoid bias.
RECOMMENDED ASSURANCE PRACTICES
We categorize assurance practices into two broad areas: design and implementation. Design includes team organization, capability, and management support. Implementation includes the review process and feedback, ownership and accountability of the forecasts, linkage of forecasts to economic models, and distribution of learnings from the experience.
The assurance team should report directly to the head of exploration and serve all exploration teams as required. The most effective assurance teams have independent dedicated, multidisciplinary core members who interact frequently with technical teams and management. Corporate assurance guidelines can drive consistency by defining terminology, the types and purpose of various reviews, and the logistics to arrange and conduct a review. They should describe what work products are expected from the technical team for these reviews, and in return, the nature and scope of documentation that the assurance team will provide postreview.
Assurance teams formed on an ad hoc basis are less effective, because other assignments and distractions impair evaluation consistency. In some companies, the head of the geoscience and reservoir engineering functions are responsible for assurance, which can present conflicts with other duties.
Regional business unit–based assurance teams are subject to influence by local management, creating a conflict of interest for the technical team and making company-wide consistency more difficult to maintain. To address that problem, Harper (2000) noted enactment of (1) technical networks linking units that discussed problems and shared best practices; (2) rigorous peer assistance sessions in which the prospect team invited individuals from other units to review, advise, and challenge; and (3) incentives for unit managers to forgo funds for their own projects to fund better projects in other units.
Assurance Team Capabilities
The size and composition of the assurance team will vary by company size. The team should include individuals who have not only broad knowledge across several disciplines but also people with deep knowledge in critical exploration specialties. A major positive change in the last decade revealed by the 2019 survey of assurance coordinators (see Appendix) was that 63% of responding assurance teams have a reservoir engineer to assist with the hydrocarbon recovery and development planning inputs.
All team members must have a solid technical foundation in opportunity generation and characterization. The assurance team manager has the primary responsibility for succession planning within the assurance team. Given the long cycle of most E&P projects and reduced exploration budgets, gaining reliable and practical experience can take up to a decade. Individual rotations must be planned across a multiyear horizon to ensure both the consistency and the continuity of the process. An assurance team assignment extended over several budget cycles enables personal calibration across the life cycle of prospects.
While technical excellence and experience are fundamental prerequisites, assurance team members should possess important nontechnical skills. Individuals must demonstrate the ability to listen before providing any challenge. Assurance team members should periodically take steps to understand and mitigate their own personal biases. To gauge their effectiveness, the assurance team can periodically survey the technical teams and managers (Bond, 2008). Ideally, technical teams would see the assurance team as a supporting partner to their evaluations.
Boyd (2019) emphatically stated that assurance requires management support to be effective. Assurance teams have been labeled by some as the “risk police,” and McMaster (2008) explored the causes of this derogatory label. He found that some business units allowed prospects to proceed without assurance reviews, and a lack of trust that each business unit would be treated equally in terms of funding. To ensure that assurance is a necessary component of portfolio management, senior management should refuse to consider prospects unless they have been properly assured. A desired state is summarized in the quote, “Assurance provides the underpinning of everything we do in decision-making and portfolio management” (B. Ryan, 2017, personal communication). Building the case for management support inevitably falls to the assurance team and how well they can demonstrate exploration improvement once implemented.
The assurance process must be sufficiently flexible to address various scales of assessments and externally imposed deadlines. Scheduling assurance reviews is the responsibility of the technical team as their projects approach key decisions. Figure 4 shows the recommended interaction of the assurance and technical teams in a typical exploration project. Given the long life cycle of most E&P projects, the assurance team is most effective when involved at several points as projects mature and in the critical postdrill review. Framing sessions provide early engagement for regional perspective and analog selection. Large complex opportunities typically require several reviews, whereas a small opportunity may only need a single review (Figure 5). The more familiar the assurance team is with an opportunity, the less likely conservatism will manifest itself in their evaluation.
Effective feedback is structured, consistent, timely, and actionable (Phoel, 2009). The assurance team must provide recommended actions, clearly stated in written feedback, ideally delivered within a week of the review. The feedback must include a session summary, agreed-to or recommended revisions of the volumetric distributions, associated chance factors, and suggestions for further technical work with supporting justification. Including the assurance team leader in decision sessions can facilitate discussion on significant subsurface issues. Written documentation is critical to tracking predictive performance, particularly because years may separate the final agreed-to predictions and the drilling result.
Ownership and Accountability
Accountability for the predictions of resource and geologic chance of success is generally placed with the technical team or manager after having carefully considered the assurance team’s view. Occasionally, the technical team will disagree with the assurance team’s assessment, or it may not be possible to implement the recommendations of the assurance team due to external factors. In these cases, the technical team’s numbers are used for drilling authorization, clearly documenting differences with the assurance team to the decision maker. An alternative approach used in some companies, but not widely adopted, is for the technical teams to present their work to the assurance team. The assurance team then generates and is accountable for the final resource and chance numbers. The technical team’s business unit remains accountable for economic assessment (Citron et al., 2017).
Linkage to Value
The assurance team’s remit may extend into appraisal and development discussions. Many subsurface and above-ground engineering issues can affect project economics of resource and chance assessments (Schneider, 2016). Drivers in the valuation of an opportunity that should similarly benefit from a review process include productive rates, well counts, costs, and investment scheduling. This commercial review should not be part of the resource and chance assurance process. In the track record of large E&P projects, Merrow (2012) described a lack of continuity in project leadership as a contributing cause of poor performance. Ensuring continuity of assurance from exploration through appraisal could help offset this important “handoff” issue. The assurance team can provide management with useful insights into a company’s prospect inventory based on their broad exposure to the opportunity base. They can also identify unique characteristics of prospects, such as dependency between prospects residing in the same play.
Performance tracking is the method by which companies evaluate the accuracy of their technical assessments. Citron (2021) provides a historical perspective on the method, and previous sections in this paper have cited several examples that illustrate how performance tracking has aided assurance. The most critical component of performance tracking is a rigorous postdrill review.
The assurance team usually takes the primary responsibility to capture and compile postdrill results. These results are compared with the predrill estimates to identify gaps in prediction accuracy and generate recommendations to close those gaps. Detailed analysis of both the dry holes and successful wells always provides lessons learned. Baird et al. (1999, p. 21) stress that, “A lesson is not learned until something changes.” A best practice is to review and share key lessons from all recent postdrill evaluations with both the technical teams and management, agreeing on what needs to change. For companies with a small annual portfolio, performance tracking over a multiyear analysis window becomes important for pattern identification.
McMaster and Carragher (2003) demonstrated the power of performance tracking as an implementation barometer (Figure 6). For a single company they documented widespread resource overestimation in each of the first 3 yr of the assurance process. As assurance progressed the next 3 yr, they recorded resource delivery within acceptable bounds set by management (shaded in Figure 6). Notably, in year 7, they illustrated a dramatic underestimation from drilling a 48-well portfolio that did not result from widespread error, but from just 2 wells. Those 2 wells shared an incorrect seismic time-to-depth conversion uncovered in the postdrill assurance review that was subsequently corrected. The improved workflow was implemented across exploration companywide. Results in year 8 fell back into acceptable bounds.
The primary objective of any exploration organization is to deliver on their predictions. Providing accurate predictions is challenging, with many companies making repeated mistakes that destroy value. Based on analysis by multiple data sources, a robust assurance process, coupled with certain quality control processes, can produce a more predictive portfolio and contribute to improved business results. A key aspect of the process is a dedicated, multidisciplinary, independent, and centrally controlled assurance team. This team can enhance the predictive accuracy of technical team forecasts and improve the decision makers’ confidence in an investment decision. Validation of technical assessments by assurance contributes significantly to consistent and predictive portfolio management.
The authors are grateful to Peter Carragher, Jim Gouveia, Paul Haryott, Jim MacKay, Bob Otis, Peter Rose, Mark Schneider, Lisa Ward, Ray Young (all with Rose & Associates), and Graeme Bagley (Westwood Global Energy Group) for their contributions. For more than a decade, over 60 assurance practitioners and members of the Rose & Associates Risk Coordinators Network have shared both successes and challenges as they establish and practice sound assurance. Their openness and honesty helped shape this paper. The authors also thank the anonymous AAPG reviewers for their constructive comments and feedback.
Marc Bond is a senior associate with Rose & Associates. He has 40 years’ international experience. Previously, he worked for BG Group, with roles including chief geophysicist and subsurface assurance manager. He manages the Rose & Associates Risk Coordinators workshops. He received his M.Sc. degree in geophysics from the Colorado School of Mines and his B.Sc. degree in geology from the University of California, Santa Cruz.
Gary P. Citron is a senior associate with Rose & Associates. In 2008, he cofounded the Rose & Associates Risk Coordinators Workshop, which continues today. From 1994 to 1998, he served on Amoco’s assurance team after various technical and managerial assignments. He received his B.S. degree in geology from the University at Buffalo, and his Ph.D. in geology from Cornell University.
Doug Weaver is a partner with Rose & Associates. He has held a variety of exploration staff and management positions with Chevron and Texaco, including Chevron’s assurance team. Along with Marc Bond, he coordinates the Rose & Associates risk coordinators workshops. He has a B.Sc. degree in engineering from the University of Illinois and is a licensed petroleum engineer.
APPENDIX: THE CURRENT STATE OF ASSURANCE TEAM CHARACTERISTICS AND PRACTICES
A 2019 survey of assurance practitioners (called the Risk Coordinators Network) provided a recent snapshot of how companies design and implement assurance. Thirty-six international E&P companies of varying size, including government agencies, took part. Figure 7 summarizes the results. The most revealing outcome was that the pre- versus postwell analyses conducted by assurance teams were not always shared with the technical staff, impeding future improvement.
Survey responses identified the top five areas in assurance working well, and the top five areas for improvement are as follows.
The fully stated survey questions (letter in parentheses refers to the respective plot) follow.
(A) How many dedicated members are currently assigned to the team?
(B) What is the average experience level of the team?
(C) What percentage of the dedicated team members are full-time, part-time, or ad hoc?
(D) To whom does the lead of your assurance team report?
(E) Approximately how many reviews per year do you hold?
(F) Who has ownership of the final assurance numbers?
(G) On average (during the past 12 months), how long are assurance reviews?
(H) Does the assurance team have direct input into the economics and commercial applications of the assured opportunity?
(I) Do you provide an annual review of the compiled postwell analysis results and learnings to management?
(J) What are the disciplines of the various dedicated team members?
(K) What is the scope of your assurance team? E = Exploration, A = Appraisal, D = Development
(L) How many times on average (during the past 12 months) do you review an opportunity in advance of the presentation for authorization?
(M) Do you provide written summary notes postreview?
(N) Is your assurance team centrally or regionally based (or a hybrid of the two)?