A three-year European Commission project entitled performance assessment methodologies in application to guide the development of the safety case (PAMINA) was conducted in the period 2006–2009 and brought together 27 organizations from ten European countries, including the Nuclear Decommissioning Authority and Galson Sciences Ltd from the UK. The PAMINA project had the aim of improving and developing a common understanding of performance assessment (PA) methodologies for disposal concepts for spent fuel and other long-lived radioactive wastes in a range of geological environments.

Work undertaken within PAMINA focussed on four areas: (1) review of PA methodologies in participating organizations; (2) treatment of uncertainty in PA and the safety case; (3) other methodological advancements in PA; and (4) relevance of advanced PA approaches to practical cases.

The state of development of a radioactive waste disposal programme has a strong influence on the type of safety case and supporting PA that is produced. A range of PA methodologies has been developed by different waste management organizations. This paper presents a selection of conclusions from the PAMINA project, in the context of general understanding developed on what would constitute an acceptable safety case for a geological disposal facility, and outlines areas for further development.


A safety case for a geological disposal facility (GDF) is a set of claims concerning the safety of the disposal of radioactive waste, substantiated by a structured collection of arguments and evidence. In the UK, when referring to safety of the environment and members of the public, such a case is described as an environmental safety case (ESC). Within the safety case, the performance of the facility against quantitative safety standards is evaluated through calculations. A quantitative (environmental) safety assessment involves developing an understanding of how, and under what circumstances, contaminants might be released from a GDF, how likely such releases are, and what the potential radiological or other consequences of such releases could be to humans and the environment. Importantly, it is necessary to understand how the geological characteristics of the site and the components of the design will evolve and function, and document the uncertainties associated with the assessment and their potential consequences. The term performance assessment (PA) is used more generally to refer to analyses of the performance of the overall geological disposal system or of particular subsystems.

Some key features of a safety case are illustrated in Fig. 1. A safety case needs to bring together and effectively integrate a wide range of safety arguments and analyses. A safety case requires a siting and design strategy, an assessment strategy and a management strategy. The siting and design strategy describes how the system will provide safety through the use of multiple engineered and natural barriers. The assessment strategy describes how the requisite level of safety can be demonstrated. The assessment strategy needs to make use of both quantitative and qualitative lines of reasoning. Both strategies should not rely unduly on any single component, whether that is a physical component of the system (such as a particular engineered barrier) or a single element of the assessment strategy (such as numerical modelling). The management strategy needs to ensure that siting and design and assessment strategies are implemented with the appropriate degree of coordination and quality.

The whole of the safety case must be underpinned by the results from research and development (R&D) studies, design and site characterization, and demonstration of how regulatory requirements and guidance have been or will be met. The safety case will need to be developed in a staged manner, consistent with a staged approach to GDF conceptual development, feasibility studies, site selection and characterization, licensing, construction, operational testing, full-scale operation and closure. The safety case will need to be informed at each development stage by dialogue with the regulators and other stakeholders.

The parts of the safety case considered within the European Commission (EC) PAMINA project are illustrated using italics in Fig. 1 (namely: safety functions; safety arguments and the use of PA modelling, complementary performance indicators, and optimization/best available techniques (BAT); regulatory dialogue; and wider communication). This paper provides an introduction to the PAMINA project, considers the treatment of uncertainty in the safety case, summarizes some other specific outcomes and conclusions of the project, and provides some ideas on areas for further development.

Performance assessment methodologies in application to guide the development of the safety case (PAMINA)

The PAMINA project was part of the EC Sixth Framework Research Programme, and ran from 1 October 2006 to 30 September 2009. It brought together 27 organizations from ten European countries, and included one EC Joint Research Centre. In addition, there were several associated groups, from four other countries, which extended the reach of the project and brought in additional experience, from other European countries, North America and Asia.

The work within PAMINA was organized into four research and technology development components (RTDCs) having the following main aims:

RTDC-1: To evaluate the state of the art by undertaking a comprehensive review of PA methodologies and experience in participating organizations.

RTDC-2: To establish a framework and methodology for the treatment of uncertainty in PA and safety case development, and to document good practice.

RTDC-3: To develop and improve particular PA methods and tools.

RTDC-4: To conduct collaborative PA exercises designed to understand the potential implications of undertaking modelling at different levels of process (simplified, detailed) and geometric (1D, 2D) complexity.

Each RTDC consisted of a number of interrelated work packages. A fifth component was dedicated to training, knowledge management and dissemination of results.

There were 22 publicly available milestone reports and 32 deliverable reports produced within the project. The milestone reports have been made publicly available where they are self-standing and contain work of wider interest that is not presented in sufficient detail in a deliverable report. All of these reports are available on the PAMINA internet site (www.ip-pamina.eu). The project involved some 50 person years of effort, at a total cost of about eight million euros.

The results of RTDC-1 form what is referred to as the European Handbook of Safety Assessment Methods for Geological Repositories – Part 1; the results of RTDC-2, RTDC-3 and RTDC-4 collectively form the European Handbook of Safety Assessment Methods for Geological Repositories – Part 2. The European Handbook is therefore the key output from the project. The main introduction to the project and its results is provided in the PAMINA Project Summary Report (Galson and Richardson, 2011), which is available on the project internet site.

The European Handbook Part 1 (Bailey et al., 2011) is based on reviews conducted of the state-of-the-art as of the start of the project, and is divided into the following topic areas: (1) safety functions; (2) definition and assessment of scenarios; (3) safety indicators and performance/function indicators; (4) uncertainty management and uncertainty analysis; (5) safety strategy; (6) analysis of system evolution; (7) sensitivity analysis; (8) modelling strategy; (9) human intrusion; (10) biosphere modelling; and (11) criteria for data selection/input. The other RTDCs take forward work in all of these topic areas.

Consideration of uncertainty in the safety case

At every stage of development the safety case will need to consider uncertainties, and will need an overall strategy for the management of uncertainty so that the project can move forward even as uncertainties are still being resolved. Such strategies will need to take account of uncertainties that can be explicitly included in PA models and those that are excluded from the quantitative PA modelling.

Overall strategies for management of uncertainties considered outside the PA generally contain elements designed to:

  1. Show that the uncertainty is unimportant to safety, for example because safety is controlled by other processes.

  2. Use more qualitative or semi-quantitative assessment methods to rule out uncertain events based on low probability of occurrence or because other consequences, were the event to occur, would be far more significant to society, for example, direct strike by a large meteorite. Qualitative arguments can be particularly important in considering events far removed in space and time from the original emplacement of waste in the GDF, and where there are very large uncertainties associated with the quantitative assessments.

  3. Optimize the design of the system via the use of conservative engineering design principles, such that the uncertainty becomes less important. An example of this approach is provided by the Finnish and Swedish programmes, where the engineered barriers used in the KBS-3 disposal concept are extremely robust, so that greater uncertainty can be tolerated with respect to performance of the far field and biosphere.

  4. Ensure appropriate quality assurance and management systems are in place for all aspects of GDF development. Most national programmes have applied custom-designed or internationally accredited QA procedures to their operations.

Overall strategies for treatment of uncertainties included in a PA generally contain elements designed to: (1) address them explicitly in the PA model; (2) bound the uncertainty and show that the bounding case still provides an acceptable level of safety; (3) agree a stylized approach to the uncertainty, for example as often done for assessment of future human actions.

There is consensus on how uncertainties considered in PAs should be classified and on the nature of uncertainties, although this consensus can be masked by variations in terminology and differences in the way uncertainties are treated in programmes. Uncertainties in PAs are generally classified as:

  1. Uncertainties arising from an incomplete knowledge or lack of understanding of the behaviour of engineered systems, physical processes, site characteristics and their representation using simplified models and computer codes. This type of uncertainty is often described as model uncertainty. It includes uncertainties that arise from the modelling process, including assumptions associated with the reduction of complex process models to simplified or stylized conceptual models for PA purposes, assumptions associated with the representation of conceptual models in mathematical form, and the inexact implementation of mathematical models in numerical form and in computer codes.

  2. Uncertainties associated with the values of the parameters that are used in the implemented models. They are variously described as parameter, or data uncertainties.

  3. Uncertainties associated with the possible occurrence of features, events and processes (FEPs) external to the disposal system that may impact the natural or engineered parts of the disposal system over time. These are usually described as scenario or system uncertainties.

All three classes of uncertainty (model, data and scenario) are related to each other. This means that particular uncertainties can be handled in different ways, and might be considered as model, parameter or scenario uncertainties within any single iteration of a PA/safety case, depending on programmatic decisions (e.g. on how to best to implement PA calculations or to communicate results). For example, variability in the geological environment could be regarded as a model uncertainty (and addressed by considering different conceptual models), or as a parameter uncertainty (and addressed by assigning appropriate parameter ranges to the geological properties).

The classification system for uncertainties given above essentially arises from the way the PA is implemented, and says little about the nature of the uncertainties. With respect to nature, a useful distinction can be made between epistemic and aleatory uncertainties. Epistemic uncertainties are knowledge-based, and therefore, reducible by nature. Aleatory uncertainties, on the other hand, are random in nature and are irreducible. All three classes of uncertainty as defined above contain elements that are epistemic and aleatory, although it may be generally true that scenario uncertainties contain a larger element of aleatory uncertainty than the other two groups.

There are strategies for treating each of these kinds of uncertainty in a PA, and within PAMINA these strategies were examined in detail and research undertaken to refine and further develop approaches. This work is summarized in Crawford and Galson (2009).

Selected conclusions from PAMINA

Key conclusions from PAMINA, which can serve to focus future research and development in the area of PA and the safety case, include:

  1. Whereas in the past, safety case development placed a lot of emphasis on comparison between safety assessment calculation results and dose/risk criteria set by the regulator, recent safety cases have used a broader range of performance indicators and safety arguments: BAT, optimization, safety functions and alternative safety and performance indicators are increasingly being used as additional arguments in a safety case in support of compliance with the regulatory dose/risk criteria and to build confidence in the long-term safety and the robustness of GDF design options. Doses and risks remain as primary safety indicators, but it is understood that over long timescales such calculations should be considered as illustrative.

  2. Calculation of a range of alternative safety and performance indicators beyond the traditional dose/risk approach can assist in demonstrating safety, understanding of subsystem performance, and building confidence in the multi-barrier approach and optimization decisions. It can also assist in wider communication of the safety case when addressing both technical and lay audiences. This does not remove the need to provide detailed calculations to regulatory authorities for comparison to regulatory dose/risk performance measures, but the use of alternative indicators provides a useful adjunct. Further development and application of these approaches would be beneficial.

  3. The main focus of safety assessment remains an evaluation of radiological impacts on humans, but there is an increasing recognition of a need for consideration of the potential impacts on non-human biota, as well as the potential impacts of chemotoxic elements in the wastes.

  4. As programmes mature, safety case development is being driven increasingly by two elements. First, the requirements for staged updating at key programme decision stages, where decisions are likely to be based on a much wider range of factors than purely safety arguments, such as the need to demonstrate optimization at every stage of development and the use of BAT. Second, the development and implementation of an assessment approach based on the use of safety functions that tie into the multi-barrier approach is being increasingly used as a means to structure assessments and to communicate the outputs.

  5. Catalogues of FEPs describing all of the possible influences on the disposal system are seen as useful in driving or auditing the development of expected evolution (or base case) scenarios and altered evolution (or variant) scenarios for use in PA. Scenarios are increasingly being developed by consideration of how particular FEPs could affect the safety functions of a particular disposal system. However, there is a view that the FEPs to be considered have been largely identified through structured FEP elicitation exercises conducted in many countries in the 1990s, and there has been little effort to develop fundamentally new FEP databases since then.

  6. The main consideration in the assignment of probabilities to scenario-forming FEPs is safety case robustness and credibility. Where statistical information is available, this should be used. Otherwise, probabilities should be assigned on a cautious basis and should be avoided where regulatory guidance provides for this; where insufficient information is available; where assessment outcomes do not depend on this probability; or where siting has already explicitly considered the uncertainty and there is nothing that can be done to reduce the probability further. Where formal expert elicitation is used to define probabilities, it is important to record the experts' thinking and to identify any factors that could affect probability estimates, in order to demonstrate transparency in attributing probabilities to particular parameters or events. Use of formal methods may be justified where safety case outcomes rely significantly on probability estimates. Robustness and credibility may also be enhanced by careful explanation that most probabilities are ‘degrees of belief’ or ‘weightings’, rather than formal mathematical probabilities. Such a treatment means that it is permissible to assign scenario weightings that total more than one. This allows for a robust treatment of scenario uncertainty in the PA, but would not be consistent with a purely mathematical treatment of scenario probabilites.

  7. As there is little scientific basis for predicting the nature or probability of human actions in the far future, the safety case for a GDF should focus on the potential consequences of inadvertent intrusion using one or more stylized scenarios. In contrast to the assessment of naturally occurring FEPs, such analyses need not aim for comprehensiveness. The range of possible future human actions is large, and it is more appropriate to evaluate the resilience of the disposal system design to stylized events. In a number of countries, regulations have specific requirements on how inadvertent human intrusion should be treated in assessments.

  8. There is significant interest in developing more complex models to represent the different components of the disposal system as programmes mature, in order to demonstrate adequate knowledge and capability to evaluate system behaviour over time and to assist with design optimization. Comparisons between models having greater and lesser geometric and process complexity have demonstrated that in the early stages of a GDF development process, simplified models can be successfully used to provide an indication of where more detailed investigations are required. As the programme matures, more complex models are likely to become available. If the results obtained using a complex model with many parameters can be reproduced using a simple model with a few parameters, it is clear that the key processes and parameters (those included in the simplified model) have been identified and the system is reasonably well understood. This would be a strong argument in the safety case.

  9. Whether conservative or best-estimate assumptions and parameter values are used in a PA, and whether deterministic or probabilistic calculation methods are used, these should be based on a transparent use of expert judgement. When combined with a clear audit trail, this will allow regulators and other interested stakeholders to better understand the potential impact on safety posed by model, parameter and/or scenario uncertainties, and the way in which these have been addressed. Guidance has been developed within PAMINA on good practice for formal expert elicitation and the treatment of parameter and model uncertainties, the selective use of which can help introduce a higher level of consistency and confidence in assessment outcomes and the safety case.

  10. Sensitivity analysis is an important tool in understanding the impacts of particular model inputs on the overall safety of the disposal system, and allows effort and investigations to focus on those parameters, models and scenarios that have the greatest potential impacts on safety. Comparisons of sensitivity analysis approaches using both synthetic problems and real data from ongoing site-specific investigations have shown that the current level of capability amongst those working in the field is high, and adds to the confidence that suitable models and analytical approaches are available. Guidance has been provided within PAMINA on what techniques are most suitable for use in particular circumstances. In actual assessments, approaches to sensitivity analysis tend to follow well established methods, and work within PAMINA has also shown that the use of different techniques can lead to similar conclusions on parameter sensitivity.

  11. Spatial variability of parameter values can have considerable impact on the understanding of subsystem performance and the safety functions ascribed to subsystems, such as mechanical stability and the ability of the geosphere to retard migrating radionuclides. There is a need for further work concerning the difficulties of transforming individual measurements of safety-related parameters, such as fluid flow rates and hydraulic conductivity, into parameter values that can be used with greater justification in large-scale radionuclide migration models. Examination within PAMINA of a new approach to simulate radionuclide transport as a sequence of particle transfer rates (continuous time random walk) has indicated that this could offer a powerful and effective means to quantify radionuclide transport in a wide range of porous and fractured media.

  12. The maturity and complexity of biosphere modelling approaches and dose assessment strategies differs between organizations in different countries, mainly due to differences in national regulatory frameworks and differences in the maturity/timing of the programmes. Issues associated with the biosphere modelling required for long-term assessments of radioactive waste disposal have been dealt with in greater detail in other international collaborative projects (e.g. BIOPROTA, www.bioprota.org).

  13. The PAMINA project included significant work on the regulatory perspective to PA and the safety case. Some of the high-level conclusions are that dose-based regulatory criteria should avoid language that discourages a developer/operator from exploring the full range of uncertainty owing to a concern that some calculations might yield results exceeding the criteria. Risk-based criteria should not be limited to requesting the presentation of mean values, but should encourage the developer/operator to discuss and present the entire range of uncertainty. Given that long-term calculated doses are interpreted more as illustrative performance measures, the validity of basing regulatory decisions largely on the use of a dose limit for the long term is questionable. This line of thinking has led to significant regulatory redevelopment in a number of countries over the last 10 or so years. Second-generation regulations for GDFs take explicit account of the wider understanding developed within the safety case and the importance of concepts such as optimization, BAT, and safety functions in driving decision making. However, given the long timescales of GDF development programmes, regulators will continue to learn, and future regulatory guidance will increasingly be informed by the national GDF development programmes.

Where are we now and where next?

There is an increasing databank of national safety assessments, built up over the last 25 years or so. About 20 years ago the Radioactive Waste Management Commmittee of the OECD Nuclear Energy Agency, the International Radioactive Waste Management Advisory Committee of the International Atomic Energy Agency, and the EC's Experts for the Community Plan of Action in the Field of Radioactive Waste (OECD Nuclear Energy Agency, 1991) were collectively able to:

“[confirm] that safety assessment methods are available today to evaluate adequately the potential long-term radiological impacts of a carefully designed radioactive waste disposal system on humans and the environment.”

and to:

“[consider] that appropriate use of safety assessment methods, coupled with sufficient information from proposed disposal sites, can provide the technical basis to decide whether specific disposal systems would offer to society a satisfactory level of safety for both current and future generations.”

There is good knowledge of the sources of uncertainty in PA and the safety case, and how they can be managed through a multi-factor safety case and multiple lines of both qualitative and quantitative reasoning, as evidenced in PAMINA RTDC-2 (Crawford and Galson, 2009). There is also a good understanding that expert judgement runs through all steps of a PA and the safety case, and that systematic approaches are needed to develop the PA and the wider safety case, as evidenced in PAMINA RTDC-1 Handbook Part 1 (Bailey et al., 2011). Finally, there is good understanding of the issues involved in regulating a project that requires safety to be considered over many thousands of years.

The PA and safety case topics that would benefit from further development include the following:

  1. The management of PA and integration of PA activities with other parts of the disposal programme as the programme matures. In particular, there is a need to better understand (i) how PA methods can be used to support optimization of design as a programme moves closer to actual implementation; (ii) how PA can be used as a tool to help inform and prioritize investigation and R&D studies; (iii) the appropriate balance between quantitative PA methods and qualitative arguments in a safety case; and (iv) how to maintain traceability of how and why a PA and the safety case evolve, as disposal programmes move forward over many decades.

  2. The communication of PA and the safety case to stakeholders having different degrees of understanding and/or different frameworks for understanding about long-term safety issues.

  3. Further refinements to PA tools, for example increasing use of fast-running system assessment models, supported by more detailed component models, to assist decision making with such issues as optimization, disposal layout, waste packaging proposals, and development of waste acceptance criteria. There has also been renewed interest internationally in the use and development of total system probabilistic assessment approaches, as an adjunct to mainly deterministic approaches, to better understand model sensitivities.

Further work has already been undertaken at international level [e.g. within the context of the NEA methods for safety assessment (MESA) project] to continue the dialogue on some of these issues.

Finally, we note that even as more data and understanding accrue, there will always be uncertainties that remain to be managed in the safety case, particularly those that are present over long timescales. There will be a need to demonstrate that such residual uncertainties are unimportant. However, the regulator will always have the job of making a decision in the face of these uncertainties. Further regulatory development and guidance can be expected on PA and the safety case, as programmes mature and as regulators learn along with the teams developing the PA/safety case.


The authors acknowledge the support of the EC under contract FP6-036404, and co-funding received from NDA RWMD (UK), ONDRAF/NIRAS (Belgium), Nagra (Switzerland) and SSM (Sweden).

P1Freely available online through the publisher-supported open access option.