P1Freely available online through the publisher-supported open access option.

Abstract

Programmes for the geological disposal of radioactive wastes are by nature extremely complex. A structured approach for making and documenting varied kinds of decisions is required to support programme design and implementation. At each programme stage, the decision-making process must be able to identify and justify key priorities for work, to reduce uncertainties.

To support structured decision-making evidence support logic (ESL) has been developed and applied to varied complex projects, nationally and internationally, in several industries. Evidence support logic involves breaking down a hypothesis that informs a decision into a hierarchical ‘decision tree’. Examples of hypotheses are ‘the geology associated with site x will provide sufficient disposal capacity’, ‘container x will contain waste form y for z years’ and ‘the engineered barrier system will provide the required safety functions’. Independent evaluations of confidence ‘for’ and ‘against’ bottom-level hypotheses allow the level of remaining uncertainty (or conflict) to be recognized explicitly, and the overall confidence (and uncertainty) relevant to the overall decision, and key sensitivities, to be represented clearly and succinctly.

Thus ESL can help (1) break down decisions into a manageable and logical structure, assisting clear presentation; (2) identify key uncertainties and sensitivities to inform prioritization; and (3) test whether the outcomes of specific studies have improved confidence.

Introduction

Evidence support logic (ESL) is a technique that is designed to support decision makers and analysts in making complex judgements that are subject to uncertainty (Cui and Blockley, 1990; Bowden, 2004; Quintessa, 2007). Decisions associated with the management of radioactive waste facilities are typically informed by a wide range of factors, drawing on multiple information sources. This is true for top-level siting and programme strategy and prioritization decision making and for detailed judgements relevant to specific questions on system performance.

Assessment processes in support of such decisions can involve having to assemble and manipulate substantial amounts of evidence. Decisions are then made on the basis of judgements about the quality and reliability of that evidence and the extent to which it supports a given interpretation. Moreover, although there may be a large volume of information relating to the decision at hand, it may be of only partial relevance, incomplete, uncertain or even conflicting.

There may be disputed interpretations of the evidence, perhaps because some people may appear to be biased by excessive reliance on a particular source in the face of contradictory, or more equivocal, evidence from elsewhere. Hence, in order to provide a justified interpretation of the available evidence, which can be audit-traced from start to finish, it is necessary to judge transparently both the quality of the data and the quality of the interpretation and modelling process. A systematic approach to the assessment of such evidence therefore allows the levels of confidence in such data to be quantified and justified.

Evidence support logic has been developed as a deliberative process for addressing questions of confidence in decisions based on uncertain evidence. The aim of ESL is to guide understanding of how the evidence combines to support confidence judgments, and to identify the amount of uncertainty or conflict associated with evidence relating to a particular decision. This understanding can be used to make a robust case for a particular decision that is subject to uncertainty, and/or to identify key areas of conflict or uncertainty that represent priorities for further investigation and analysis in order to build confidence in the overall outcomes.

Overview of evidence support logic

Evidence support logic involves breaking down a proposition into supporting lines of reasoning, until hypotheses associated with those lines of reasoning are identified at a level of detail suitable to facilitate analysis of the underpinning evidence. Judgements on confidence for and against hypotheses are made independently on the basis of available evidence. Where the confidence arising from that evidence is not complete, uncertainty remains and is explicitly represented. If there is substantial evidence both for and against, conflict in evidence may be identified. The basic judgements and opinions are recorded to provide a transparent evidence trail.

The ESL approach thus differs from classical probability theories which require evidence to be either in favour of a hypothesis, or against it. This is sometimes described as a ‘closed world’ perspective, in which evidence for and evidence against are treated as complementary concepts. In contrast, three-value logic allows for a measure of uncertainty as well, recognizing that belief in a proposition may be only partial and that some level of belief concerning the meaning of the evidence may be assigned to an uncommitted state. Uncertainties are handled as ‘intervals’ that enable the admission of a general level of uncertainty, recognizing that information may be incomplete and possibly inconsistent

Key stages in the process

Application of an ESL process involves three main steps:

  1. Development of a hierarchical logical hypothesis model to provide a common, coherent structure for assembling all the evidence that is relevant to an identified root (or top-level) hypothesis (or proposition).

  2. Parameterization of the logical model and identification of sources of evidence that contribute to arguments for and/or against sub-hypotheses in the model.

  3. Input and propagation of evidence through the logical model, using the principles of interval probability theory (Cui and Blockley, 1990) to represent uncertainty and to provide an assessment of the dependability of the overall root hypothesis.

The hierarchical model is developed by breaking down and structuring the key issues that need to be considered in evaluating confidence in the root hypothesis. In doing so the aim is to define, structure, and communicate the rationale for making the assessment. Ultimately, the model needs to link to available sources of evidence for eliciting the level of confidence in each leaf hypothesis. The confidence levels are derived from separate assessments of the evidence for and against those hypotheses.

This confidence is then combined and propagated through the model, subject to the logic and parameterization implemented within it to reflect the relative importance of each child hypothesis to the parent hypothesis it supports, and ultimately to the levels of confidence derived for the root proposition.

Typically these steps are applied within an iterative process, whereby the ESL model and outcomes are tested and developed, recording the outcomes in a code such as TESLA (Quintessa, 2007). Capturing the development of understanding in iterations of an ESL model provides a powerful approach for demonstrating progress and to support continual assessment of ongoing priorities.

Development of the logic model

The hypothesis model structure defines how assessments of evidence combine to provide overall confidence. This propagation of confidence also ensures uncertainty is appropriately represented at the top level. Analysis of the model then shows which evidence sources, or areas of uncertainty, are important. The model thus provides an excellent tool for communication of the importance of key judgements, and how important uncertainties are to overall confidence. The process thus delivers a powerful prioritization tool.

Central to the ESL methodology is a decision tree, which is a hierarchy of nodes, each of which represents a hypothesis (Fig. 1). A decision tree is typically developed and parameterized by means of a structured discussion process involving relevant experts. The logic structure utilizes concepts such as:

  1. Group logical operators. All of a group of child hypotheses might be required to provide (or disprove) confidence in a parent. The ‘ALL’ operator ensures that confidence in the parent can be no better than the ‘weakest’ of the supporting hypothesis. Conversely confidence for (or against) ‘ANY’ of the hypotheses might be enough to prove (or disprove), in which case confidence in the parent is set as equal to that for the ‘strongest’ child hypothesis.

  2. The ‘sufficiency’ of a particular hypothesis to provide confidence for or against its parent is parameterized, independent of the subsequent evaluation of evidence, and independent of any other hypotheses that support the same parent. If a group of hypotheses are all partially or fully sufficient to satisfy (or disprove) the parent, the confidence propagated will reflect this mutual support.

  3. Necessity’ whereby sufficiency values for child hypotheses are specified as above, but there is a confidence threshold associated with one or more necessary child hypotheses below which confidence in the parent is compromised, irrespective of the contributions from other lines of reasoning.

  4. In some cases evidence evaluations for hypotheses supporting the same parent share very similar judgements on evidence. In such cases, a dependency needs to be specified to avoid double-counting.

Elicitation approaches and opportunities to involve independent experts and/or stakeholders

To ensure the outputs are robust, it is essential that each of the three main stages of an ESL study are executed in a consistent and systematic way, and include input from experts and (where relevant) review by stakeholders.

In particular, structured ESL model review workshops can provide a highly efficient mechanism for incorporating independent review through exposing key judgements, with appropriate context, to challenge. This can serve to greatly enhance confidence in judgements made. Also, this approach can provide an excellent mechanism for engagement with stakeholders including regulators, thereby facilitating communication of the rationale, context and evidence, and the relative importance of uncertainties.

To remove subjectivity as far as possible, and to ensure consistency, a standardized approach is typically followed for the elicitation of the decision tree logic, its parameterization, and for evaluation of the evidence base. For example, it is important that the sufficiencies of individual child hypotheses to support their parents are identified consistently or the results of the model will be skewed. Similarly, it is important that evidence for and evidence against hypotheses is assessed independently but consistently, as this ensures a consistent representation of the remaining uncertainty.

A number of standard questions have been formulated that apply to each of these stages, and relevant linguistic scales are used to facilitate mapping of responses to numerical input values.

Analysis and visualization of outcomes

Three main approaches to visualizing outputs from an ESL analysis have been devised for use within TESLA: the tree plot, the evidence-ratio plot and the sensitivity (or tornado) plot (Quintessa, 2007).

The tree plot (Fig. 1) provides an overview of the logical hypothesis model, showing the key contributions to confidence from different lines of reasoning, and the extent of remaining uncertainty or conflict arising at each level of the model. It presents the key logic and evidence in a structured way that typically allows the entire rationale for a complex decision to be laid out on a single sheet of paper.

The evidence-ratio plot (Fig. 2) provides a visual presentation of the distribution of confidence in evidence relating to each leaf hypothesis in the logical hierarchical model (or sub-element of the model). The horizontal axis indicates the percentage uncertainty in the evidence, or (in other words) that fraction of the total available belief that is assigned to an uncommitted state. An increasing negative value along the horizontal axis represents the existence of increasing conflict in the evidence.

The vertical axis indicates the ratio of evidence for to evidence against associated with each leaf hypothesis. Any evidence values of zero are converted to a minimum value of 0.01. This results in a possible range of between 0.01 and 100. The values are then plotted using a logarithmic scale on the vertical ratio axis.

Values plotted above the horizontal axis represent a favourable balance of evidence, indicating support for the hypothesis under consideration; those below the line represent an unfavourable balance of evidence and hence a lack of support for the hypothesis. Regions representing greater than 50% evidence for and against, respectively are shaded on Fig. 2, providing a visual guide to the extent of support that is judged to exist. The region to the left of the vertical axis reflects areas of conflicting evidence.

It can be informative to plot confidence in the main proposition associated with the ESL model on the same diagram as that associated with each leaf hypothesis in the logical model. This can provide a strong visual indicator of the potential implications of bias, arising (e.g. if there are outliers that overly influence the results that may be overly influenced by subjective opinion). By contrast, where full account is taken of the balance of evidence, including the possible weight of contradictory evidence and residual uncertainty, the true evidential support for the top-level proposition can be clearly visualized.

The tornado plot (or sensitivity plot) identifies those regions where small changes in confidence in the underlying evidence values (i.e. reducing the uncertainty) have the greatest impact on the overall result. The tornado is implemented in TESLA by temporarily incrementing by a marginal amount the evidence values of each hypothesis in turn, noting the change in evidence values of the top hypothesis. The impact is thus defined by the ratio of the change in confidence associated with the main proposition to the change in evidence for the leaf hypothesis.

The impact on overall confidence associated with each piece of leaf hypothesis evidence is converted to a percentage value and plotted as horizontal bars. The hypotheses are then plotted in descending order of total impact. An example is illustrated in Fig. 3.

Example applications

Evidence support logic has been used nationally and internationally to support decisions ranging from (for example) major investment decisions in the oil and gas and carbon capture and storage industries, to management strategy decisions for legacy nuclear facilities in the UK and the USA, and radioactive waste management and assessment decisions in Japan.

Specific examples of ESL application include:

  1. Evaluation of potential siting options for underground CO2 storage facilities, supporting major investment decisions, and advising on priorities for investigation and research (James et al., 2010).

  2. Evaluating whether potential storage sites for CO2 are likely to be suitable at an early state in site qualification (Det Norske Veritas, 2010).

  3. Assessing whether potential investigation areas for repository sites are sufficiently promising to warrant further investigation (Toida et al., 2008).

  4. Evaluating confidence in safety strategies and related key performance assessment parameters for historic radioactive waste facilities [e.g. for the wastes stored in buried single and double shelled tanks at Hanford, Washington State (Egan et al., 2009)].

  5. Evaluating whether different fracture sets are hydraulically conductive (Tsuchi et al., 2003).

  6. Judging the quality of geochemical data obtained from deep groundwater samples (Mizuno, 2007).

  7. Exploring the role of paleohydrogeology in long-term performance assessment for deep disposal (Egan and Bowden, 2004).

Potential applications of ESL for geological disposal in the UK

Overview

A number of aspects of the UK geological disposal programme could benefit from support by ESL analyses. Hypotheses that could be investigated include, for example:

  1. the geology associated with site x will provide sufficient disposal capacity;

  2. the geological disposal facility at site x will meet as-designed performance requirements;

  3. container x will contain waste form y for z years; and

  4. the engineered barrier system will provide required safety functions.

The process of development of an ESL model for hypotheses such as these would provide a mechanism for structuring and presenting the logic involved in related decisions. It would facilitate identification of key evidence sources and assessments of their value, and allow the implications of uncertainties to be assessed and communicated. The outcomes would provide a tool to help ensure a common understanding of the issues involved across the UK industry, including regulators. The model would then provide a tool for monitoring and assessing progress and communicating the impact of updates in understanding.

In addition to the direct value of assessing and justifying the confidence that can be placed in such propositions on the basis of logic and evidence, the model could provide a valuable prioritization tool, as the importance of different lines of reasoning and associated uncertainties can be utilized to help evaluate and justify research priorities. Such a model could also be used to track progress; successive updates to the model in response to improvements in the evidence base (e.g. as a result of research or characterization activities) would be recognized through corresponding reductions in white space in the model with time. The model would also provide a mechanism to record the development of the underpinning audit trail (organization of which is a key function of the TESLA software).

Example 1: potential application to MRWS stage 4

A simplified hypothetical example decision tree illustrating how ESL could be applied throughout the managing radioactive wastes safely (MRWS) process (Department for Environment Fisheries and Rural Affairs et al., 2008) is shown in Figs 4 and 5.

During stage 4 (desk-based studies) of MRWS there will be large uncertainties about many aspects of a site. These uncertainties are represented by the hypothetical white space in the left-hand tree in Fig. 4. During stage 5 (surface-based investigations) more information will be available and hence there will be reduced uncertainty (right-hand tree, Fig. 4). However, in this hypothetical example, some evidence against the site has emerged, which is represented by a red field hypothesis 1.3 (that borehole data are consistent with there being an adequate rock volume for a repository).

At the same time, there is now more uncertainty concerning hypothesis 2.3 (that investigation techniques can be deployed on necessary timescales) and some evidence against this hypothesis as might occur if.planning procedures take longer than expected.

The hypothetical tree highlights that this latter development is a priority for resolution, because it dominates the highest-level decision. However, once a resolution is found, a priority will be to remove the confidence against hypothesis 1.3 (red space), for example by changing the design/layout of the repository.

Figure 5 contains the same information as Fig. 4, but shows how the two sets of confidence values can be represented on the same ESL diagram to help illustrate the impacts of improvement of the evidence base with time.

Example 2: potential application for assessing confidence in meeting geological disposal facility design performance requirements, and identifying research required

Figure 6 shows a simplified hypothetical ESL decision tree for assessing the extent to which there is confidence that implementation of a geological disposal facility at a particular site will achieve performance consistent with design requirements. Such a tree can show how high-level confidence depends on underpinning evidence sources. Where uncertainties in evidence are shown to be important to overall confidence, the analysis will inform the priorities for research and characterization facilities. It could also be used to guide optimization studies; using the same tree to evaluate evidence for and against performance of different design options (for example) will provide a mechanism for assessing the relative potential benefits of such options.

The hypothetical tree assesses confidence in meeting design requirements for each component of a disposal facility, mapped to the engineering and performance design functions required for safety. The logic employed means that interactions between components of the facility can be assessed without double-counting confidence at the top level. Note that hypothetical failure of a component would not necessarily mean a safety case could not be made for such a facility; it may be that there is sufficient confidence in performance in the other barriers such that failure of one to meet design requirements would not lead to an overall compromise in safety. However a different top-level hypothesis and underpinning model would be required to make that case.

In the tree shown, certain lines of reasoning have been expanded to show the types of evidence might be relevant. For the example parameter set shown, there would be substantial confidence that the design requirements will be met, but there is non-trivial remaining uncertainty. In this example, much of the uncertainty can be traced back to limitations in the understanding of waste container corrosion; thus research in this area would logically be prioritized.

Summary

Evidence support logic is a powerful approach for: (1) breaking down decisions into a manageable and logical structure, assisting clear presentation of the elements of a complex decision; (2) identifying key uncertainties and sensitivities to inform prioritization of future work; and (3) testing whether the outcomes of specific studies have improved confidence.

The value of ESL has been proved through its application to support important decisions in the radioactive waste, oil and gas, and carbon capture and storage sectors. For geological disposal in the UK, ESL could be used to help structure, present and analyse the logic associated with key strategies, decisions and assessments, and to prioritize associated research and development. This would provide a valuable communication, prioritization and audit tool. Hypothetical examples have been presented based upon development of MRWS stages 4 and 5, and confidence in implementation of the geological disposal facility meeting design requirements.