An

Sa

b

a

ARRA

1

bcpcftpitacp

tItsup

0d

Nuclear Engineering and Design 242 (2012) 379– 388

Contents lists available at SciVerse ScienceDirect

Nuclear Engineering and Design

j ourna l ho me page: www.elsev ier .com/ locate /nucengdes

nalyzing the decision making process of certifying digital control systems ofuclear power plants

wu Yiha, Chin-Feng Fanb,∗

Computer Science and Information Engineering Department, Ching Yun University, TaiwanComputer Science and Engineering Department, Yuan-Ze University, Taiwan

r t i c l e i n f o

rticle history:eceived 21 November 2010eceived in revised form 3 October 2011ccepted 5 October 2011

a b s t r a c t

Safety-critical computing systems need regulators’ approval before operation. Such a permit issue pro-cess is called “certification”. Digital instrumentation and Control (I&C) certification in the nuclear domainhas always been problematic and lengthy. Thus, the certification efficiency has always been a crucial con-cern to the applicant whose business depends on the regulatory decision. However, to our knowledge,there is little basic research on this topic. This study presents a Regulatory Decision-Making Model aim-ing at analyzing the characteristics and efficiency influence factors in a generic certification process.This model is developed from a dynamic operational perspective by viewing the certification process

as an evidence–confidence conversion process. The proposed model is then applied to previous nucleardigital I&C certification experiences to successfully explain why some cases were successful and somewere troublesome. Lessons learned from these cases provide invaluable insights regarding to the reg-ulatory review activity. Furthermore, to utilize the insights obtained from the model, a prototype of acomputer-aided licensing support system has been developed to speed up review evidence preparation

egula

and manipulation; thus, r

. Introduction

Safety critical equipment needs to obtain regulators’ approvalefore operation. Such a permit issue process is called “certifi-ation” or “licensing” in safety-critical domains, such as nuclearower plants, medical devices, and aviation. The introduction ofomputer to safety-critical systems has created difficult situationsor regulators. Take a nuclear power plant for example, it needso go through a rigorous and time consuming licensing approvalrocess before its operation. Under the current one-step licens-

ng practice, the total licensing review process may take longerime (∼6–8 years) than construction process (∼4–6 years). Such

lengthy review process is not uncommon in other regulatoryompliance domains. Therefore, regulatory review efficiency hasrofound impacts on the business operation in these domains.

The process of digital I&C (Instrumentation and Control) cer-ification in nuclear industry starts when the utility presents an&C design and its associated quality evidence to the regulator;he regulator will then determine whether the presented I&C

ystem meets mandatory safety requirements prescribed in reg-lations. Thus the scope of our concern covers both the applicant’sreparation effort and the regulator’s review effort. An inefficient

∗ Corresponding author. Tel.: +886 3 4638800x2360; fax: +886 3 4638850.E-mail addresses: csfanc@saturn.yzu.edu.tw, csfanc@gmail.com (C.-F. Fan).

029-5493/$ – see front matter © 2011 Elsevier B.V. All rights reserved.oi:10.1016/j.nucengdes.2011.10.010

tory review efficiency can be further improved.© 2011 Elsevier B.V. All rights reserved.

certification process may cause unnecessary delay of the commer-cial operation of nuclear plants. Due to the enormous investment ofconstructing a nuclear power plant, any delay implies huge socialand economical loss. However, experiences showed that some digi-tal I&C certification cases were frustrating for both parties involved.It is even not too exaggerated to claim that the certification practicehas hindered the progress of introducing digital I&C techniques tonuclear power plants (Department of Energy, 2005).

Various efforts have been proposed to resolve this problem, suchas developing better regulations, etc. We propose to alleviate theI&C certification problem by extracting useful lessons learned fromprevious experiences. The rationale is that since some certifica-tion cases were successful and some were problematic; thus if wecould conduct a comparative analysis of these cases, focusing onanalyzing the reasons of their success and failure, we may identifythe critical efficiency shaping factors affecting the efficiency of reg-ulatory decision making. Then, by managing these critical factorsproperly, we may be able to avoid repeating the same problems thatwere encountered by previous troublesome cases in the future. Ourstrategy is to develop a regulatory decision making model, model-ing regulator’s decision making process, based on the informationcollected from previous digital I&C certification experiences.

The general I&C certification is performed by the regulatorsthrough safety reviews/inspections of the submitted design anddocuments to ensure the digital I&C system and its associateddevelopment process conform to regulatory quality standards.

3 ering a

Tipra

ttsWuctIoaTdmostac

2

2

2Ecb

liffgats

tltpi

oT(np

2

u

80 S. Yih, C.-F. Fan / Nuclear Engine

his study formalizes such a generic certification review model,ndependent of country-specific regulatory structures. The pro-osed model identifies model components and analyzes theirelationships and implications. Country-specific regulatory issuesre reflected in the case studies in Section 4.

To our knowledge, there is little basic research conducted athis aspect. This model is then used to identify critical factors andheir influence on certification efficiency. In this paper, first, a briefurvey of several digital I&C certification cases will be presented.

e then propose a regulatory decision making model, which issed to investigate the cause-consequence relations among certifi-ation efficiency shaping factors. The framework thus can be usedo explain why some certification case succeeded and others failed.nsight into certification efficiency will then be presented. More-ver, it is believed that advanced information technology could bepplied to speed up and streamline the licensing review process.he licensing process involves a lot of clerical work, for example,ocument preparation and syntactical evidence gathering, whichay be automatically performed by computer-aided tools. Based

n the proposed model, a prototype of a computer-aided licensingupport system has been constructed to demonstrate how to fur-her expedite regulatory decision making. In addition, our modelnd insights are general enough to be applied to other safety-criticalomputing domains.

. Related background

.1. Regulatory compliance review

There are many discussions on regulatory compliance (Sparrow,000). Most of them address this issue from management aspects.fficiency and effectiveness are the two major metrics for regulatoryompliance review. The following simple definitions are acceptedy most nuclear regulators (NEA, 2009):

Effectiveness: Do the right workEfficiency: Do the work right

NEA depicts a regulatory effectiveness model in a triangle with 4evels, from top to bottom: mission (“what is the task?”), core activ-ties, prerequisites (“How to fulfill it?”), and assessment (“Are weulfilling it?”) (NEA, 2009). The scope of this model is quite broad,rom managerial aspects, and applies to regulatory compliance ineneral. Our model will be restricted to regulatory review processnd focusing on regulatory decision making. To our knowledge,here is no similar research in modeling regulatory review fromuch an operational perspective.

Regulatory effectiveness refers to the situation that the regula-ory body has performed its job successfully to ensure an acceptableevel of safety of the licensed system. On the other hand, regula-ory efficiency concerns resource consumption in the regulatoryrocess. This paper mainly deals with regulatory review efficiency

ssues.As to regulatory compliance review tools, there are various kinds

f tools, such as those using checklists (DMID, 2010), E-submittals.here are also ontology-based tool for legal regulatory complianceGangemi et al., 2003). However, there are few licensing tools inuclear domain. Our tool is designed for nuclear regulatory com-liance review.

.2. Case studies: five licensing experiences in nuclear domain

This section surveys five licensing experiences, which will besed as case studies in this research.

nd Design 242 (2012) 379– 388

2.2.1. French Chooz B nuclear power plantP20 was the major digital control system selected by EdF for its

Chooz B nuclear power plant (Peyrouton and Pirus, 1993). It wasdesigned by Cegelec Company, which is a major French I&C providerwith abundant relevant I&C experiences. The system is a distributedmicroprocessor-based control system that resides on a redundantLAN network. A 32-bit parallel Transputer system (Guesnier et al.,1989) is chosen as its primary microprocessor. Since Transputer isdesigned for parallel applications, this system basically is runningas a parallel processing system.

This project was started in 1986. In 1990, the project wasdelayed for 2 years; it was finally given up by EdF because moredelay (estimated 4–6 years) was expected. This failed project costEdF 300 million Francs (MacLachlan, 1994a,b) and 4 years scheduledelay. The major reason for the failure is that the parallel softwareon Transputer has become too complex to be verified effectivelyand confidently. Later the system was replaced by an older productdeveloped by Controlbloc. The problem of software verification wasalleviated because the product has many year of field experience.Finally the system got certified (Appell, 1992).

2.2.2. British Sizewell B nuclear power plantSizewell B plant adopts a two-level strategy to its reactor sys-

tem, i.e., a digital Primary Protection System (PPS) backed up witha conventional secondary protection system (Nuclear EngineeringInternational, 1993). The PPS was developed by Westinghouse(WH) as one of its Eagle 21 series product. According to thestatistics published by Nuclear Electric (NE, owner of Sizewell B),Westinghouse invested 200 man-years for developing PPS and 50man-years for carrying out independent verification and valida-tion (IV&V) to assure that PPS met all related quality standards andrequirements. However, due to lack of objective software qualityand safety metrics, NE spent another 250 man-years to performa comprehensive and complicated verification and validation toimprove its confidence in PPS (Marshall and Silver, 1993).

2.2.3. Canadian Darlington nuclear power plantDarlington nuclear plant (Craigen et al., 1994) is a CANDU type

nuclear power plant developed by AECL. CANDU is among the firstnuclear plants to use computers to perform safety functions. In1982, Ontario Hydro, with the concurrence of AECB, had decidedto fully implement the decision-making logic of the shutdown sys-tem on computers. The development work began in early 1983.In early 1987, an AECB review uncovered discrepancies and raiseddoubts as to whether the software implemented the requirementscorrectly. In mid 1987, AECB hired software safety expert Dr. DavidParnas to help investigation and to recommend how to improve thesoftware. Dr. Parnas identified the areas of concerns and proposeda formal mathematical inspection in January 1989 to break thecertification impasse. The complete software requirements spec-ifications have to be rewritten in a SCR style (Heitmeyer, 2001) forinspection. This whole process completed in February 1990 whenAECB finally issued a license for full power operation. However,AECB also made a statement, requesting that the software wouldhave to be redesigned for long-term use (Craigen et al., 1994; Chunet al., 2000; Hung, 2008).

2.2.4. Japan Kashiwazaki–Kariwa 6/7 nuclear power plantsKashiwazaki–Kariwa Unit 6 and unit 7 (KK6/7) (Fukumoto et al.,

1998) is the first commercial Advanced Boiling Water Reactor(ABWR) design nuclear power plant. The basic design was jointlydeveloped by Hitachi, Toshiba and GE Nuclear Energy. The digital

I&C development process followed Japan’s regulation, JEAG 4609(1999), which in general are less complex than USNRC regulations.The I&C design was based on Problem Oriented Language (POL)which helped to make the final system reliable and verifiable. The

ering a

sbaflcf

Nsth

2

tacrmatcdiB2md

2

tcatortlbotta

2

p

U(slc

frsihtaiG

S. Yih, C.-F. Fan / Nuclear Engine

ystem validation testing was performed with the help of a PC-ased automatic testing tool which effectively reduced the timend manpower required for the validation test. A very importanteature of KK6/7 I&C project is that manufacturers have accumu-ated more than 15 years experience in handling the digital I&Components for non-safety operations. Such experience paves theoundation for the success of the project.

During certification process the regulator authority-MITI, askeduclear Power Engineering Corporation to perform a comprehen-

ive qualification testing. The testing was efficient and successful;hus, the permit was issued without lengthy negotiation as thatappened in most cases in the western countries.

.2.5. US Oconee nuclear station digital upgradeArevan’s Teleperm XS system received generic approval from

he NRC for safety applications in 2000 (Spring, 2010). Based on thispproval, Duke Energy submitted a RPS/ESFAS replacement appli-ation for the first time in 2002 that it later withdrew in 2006. Theeason for this unsuccessful licensing process is that Duke’s sub-itted documents, based on USNRC regulations, were not complete

nd detailed enough as expected by the NRC. Thus, they withdrewhe application and resubmitted its application for Oconee digitalontrol upgrades for the second time in 2008 with better preparedocuments. The company has provided supplemental supporting

nformation numerous times to show evidence for IEEE 603, BTP 14,TP 19 compliance. It was finally approved by the NRC on January9, 2010, and the installation process has begun in 2011.The $250illion upgrade is the first to receive NRC approval for a safety

igital I&C system at a nuclear power plant in the U.S.

.2.6. Case summaryGenerally speaking, the above cases showed that some digi-

al I&C certification cases suffered from high evidence preparationost (P20, Darlington, Sizewell B, Oconee), but still faced doubtsbout their potential safety performance (Darlington, Sizewell B);hus, these cases demonstrated poor certification efficiency. On thether hand, KK6/7 showed quite good certification efficiency. Theemarkable point is that the result of certification is not propor-ional to the invested effort. For example, KK6/7 spent relativelyess quality-related effort than Sizewell B, Chooz B and Darlington,ut still gained satisfactory review results from regulators. Suchbvious performance difference implies the existence of techniqueshat can improve efficiency of current certification practice. In Sec-ion 3 we will investigate what the critical efficiency shaping factorsre and identify methods that can manage these factors effectively.

.3. Related certification research

Besides the above licensing cases, there have been a number ofublished studies dealing with certifying nuclear I&C systems.

Fenton et al. (1998a), Fenton (1998b), Neil et al. (2008) at theniversity of London, proposed using Bayesian Belief Networks

BBNs) to provide diverse evidence for assessing dependability ofafety-critical systems. Their DATUM project, along with the fol-owing ESPRIT project SERENE, used BBNs to support assessors ofritical systems across Europe (Fenton et al., 1998a).

The Norwegian Halden reactor project (Gran and Dahll, 2000a)orms an international network dedicated to enhance the safety andeliability of nuclear power plants. How to certify digital controlystems of nuclear power plants has been a major research focusn the project. Both formal methods and Bayesian Belief Networksave been proposed to facilitate digital control system certifica-

ion. Sivertsen (Sivertsen, 1996) described a rigorous mathematicalpproach using an algebraic specification and an automatic provern developing a reactor safety system to assist licensing decision.ran et al. (2000b), Gran (2001) applied BBN-based approaches and

nd Design 242 (2012) 379– 388 381

related tools to supporting the review process of digital I&C systemsof nuclear power plants.

Korean nuclear industry and research institutes have investedgreat efforts in the last 20 years in the digital I&C area and haveachieved significant results. Techniques for certifying digital I&Csystems are developed by Korea Atomic Energy Research Institute(KAERI), Korea Institute of Nuclear Safety (KINS) and universities.For example, Formal method techniques such as model-checkingand petri net were developed (Jee et al., 2010; Seong, 2009) forverifying the correctness of safety-related control system of nuclearpower plants. BBN-related methods (Kang et al., 2009) were devel-oped to verify the quality of development process and productsbased on USNRC regulations (BTP 14 of NUREG 0700).

3. Development of regulatory decision making model

An evaluation framework that can investigate the performanceof certification process is presented in this section, which isfollowed by applying this framework to diagnose previous certi-fication experiences.

3.1. The nature of certification process

According to Chapter 7 Standard Review Plan (SRP) (USNRC,2007), the review guideline for nuclear industry, the most impor-tant concepts in certification process are: sufficient evidence andassurance. After a successful certification review process, a per-mit will be issued if and only if the regulator also has reasonableassurance that the submitted design complies with regulations.Therefore, the difference before and after certification process isthat, apparently, the evidence is presented to the regulator andaccepted by the regulator; pragmatically, a state of confidence isbuilt up in the regulator’s mind based on the evidence. In otherwords, the major mechanism during certification process is theconversion of the submitted evidence into the regulator’s confi-dence. Thus, we may consider the essence of certification processas an evidence–confidence conversion process.

There are many factors that can affect the efficiency of thisevidence–confidence conversion process, proper handling of thesefactors results in an efficient certification process. The followingsections will identify these factors and analyze their contributionsto the efficiency of certification process.

3.2. Regulatory decision making: an evidence–confidenceconversion process

The regulatory decision making is affected by the quality of thesubmitted documents of the applicant. Thus, certification processincludes evidence preparation process by the license applicant andthe review process by the regulator. The regulator’s review pro-cess includes a threaded review or sampling review, and through it,evidence can be converted into confidence in the safety of the sys-tem. Thus, the certification process includes evidence preparationprocess and evidence–confidence conversion process. Fig. 1 givesa schematic view of a regulatory decision making model viewedfrom an evidence–confidence conversion perspective.

The proposed model can be defined as 12 tuples:Regulatory decision making = (G, S, R, E, P, M, F, d, f1, f2, t, g)

d : G × Rp → S × E . . . developmentf1 : G × S × MA × PRE → E′ . . . evidence preparation functionf : E′ × P → R . . . prepartion resource

2 EC P

t : G × S × E′ × MR → e × RR . . . threaded reviewg : G × S × e × MR × PEF → F . . . government faith

Definition of each term is explained as follows:

382 S. Yih, C.-F. Fan / Nuclear Engineering and Design 242 (2012) 379– 388

10CFR/RG/BTP/IEEE Standards

Safety SystemDevelopmentActivities

CertificationPreparationActivities

CertificationReviewActivities

SafetyEvaluationReport

G

S

E’ RegulationComplianceEvidence

SafetySystemDesign PEF

F

MR

MAApplicantCertificationEngineer

Regulator

d

g

SelectedRegulationComplianceEvidence

e

PEC

Selection ofThreadedReview

PRE

Evaluation/EvidenceSpace

E

Evidence Preparation (vendor)Evidence Review (Regulator)

Certification Process

Development Process

Regulator

MR

f1 , f2

t

decis

1(

s

dRt(

sgetcri

t

•

•

•

Fig. 1. A regulatory

G: Governmental regulations. For nuclear industry they are0CFR codes, regulatory guides (RG), branch technical positionsBTP), technical reports, and industry standards (e.g. IEEE 603).

S: I&C System design. Complete I&C system design specificationubmitted by the applicant for certification review.

R: Resource allocation. R = Rp + RR. The term Rp represents evi-ence Preparation Resource, and RR represents evidence Reviewesource. A successful certification process aims at minimizingotal resource allocation (R =Rp + RR), and maximizing confidenceF).

E: Evidence space. e ⊆ E′ ⊆ E. The term G × S defines the statepace needed to be evaluated. E is all the evidence needs to beenerated for evaluating whether S (System) complies with G (Gov-rnment). Evidence includes documents from such QA activities asests, reviews, audits, and formal analyses, etc. In reality, the appli-ant can only prepare a subset of E, that is E′, and E′ ⊆ E under fixedesource allocation Rp. The regulator can only select a part of E′, thats e, for review. Therefore, the relation e ⊆ E′ ⊆ E holds.

P: Various Profiles/distributions shaping the performance of cer-ification activities. P ={PRE, PEC, PEF} where

PRE: REsource allocation Profile. For a particular project, it repre-sents how the total evidence preparation resource Rp is allocatedto preparing different evidence. It can be represented by percent-ages of the total presentation resources invested by each type ofQA evidence.PEC: Evidence Cost Profile. The preparation cost for different evi-dence is not equal. PEC represents the cost variance when oneprepares different types of evidence. It is a project independentdistribution.

PEF: Evidence–conFidence Profile. There are many different kindsof evidence and evidence owns different confidence conver-sion power, i.e., some evidence is more convincing than others.The evidence includes testing evidence, review evidence, IV&V

ion making model.

evidence, and formal proof evidence. Among them, for example,formal proof may generate higher confidence.

M: Proficiency Maturity level. M = {MA, MR}. It represents thedegree of proficiency of staff members performing certificationactivities. The proficiency can be roughly divided into three lev-els: novice, competent and expert. The level affects the quality ofevidence and resource consumption. MA represents the applicant’smaturity level; MR represents the regulator’s maturity level.

F: Faith/Confidence level achieved after reviewing evidence eduring the certification process.

Components of the above model are denoted in Fig. 1 along withassociated functions at each stage. These functions (d, f1, f2, t, g) areexplained below:

At the development stage, function d (d: G × Rp → S × E) refersto the fact that the developer follows governmental regulations (G)and invests preparation resources (Rp) to produce the system (S)and the associated evidence (E).

Functions f1, f2 (f1: G × S × MA × PRE → E′; f2: E′ × PRE → RP) mod-els evidence obtained and resource consumed at the vendor’sevidence preparation stage. Function f1 indicates that evidencepreparation needs to comply with governmental regulations(G), depend on system design (S), utilize the applicant’s pro-ficiency maturity (MA) to perform QA activities, and rely onresource–evidence profile (PRE) to generate evidence (E′), whichis a subset of the entire evidence (E). In the meantime, the functionf2 shows that the associated resource consumption Rp depends onthe generated evidence (E′) and the evidence–cost profile (PEC).

Regulatory reviews are usually threaded or sampling reviews. Atthis stage, functions t (t: G × S × E′ × MR → e × RR) models the evi-dence obtained (e) and resource consumption (RR) by the reviewer.

Function t indicates that the regulator’s sampling review processcomplies with governmental regulations (G) to review the I&C sys-tem design (S) based the submitted evidence (E′). Moreover, theregulator’s proficiency maturity (MR) controls the review quality.

S. Yih, C.-F. Fan / Nuclear Engineering and Design 242 (2012) 379– 388 383

Effectiveevidence

AuRTAPEC

10K

20K

40K

25K

AuRTAPRE

25%

Unit cost

P

F

AC

AuRTA AuRTA

(a) (b)

(d)(c)

GtEmr

eGpsiectu(

b

DmtR2Fetre2otira

g

E1 E3 E4 E5E2 E6

E1 E3E4 E5E2 E6

E1 E3 E5E6E4E2

E1 E3E4 E5E2 E6

(a) (b)

EvidenceConfidenceProfile

ResourceProfile

ConfidenceConfidence

% %

FF

PRE

PEF

EF

Fig. 2. An example explaining evidence profiles.

At the governmental decision making stage, function g (g: × S × e × MR × PEF → F) models confidence conversion. The regula-

ory decision making is based on G and S, as well as affected by thevidence–conFidence profile (PEF) and the regulator’s proficiencyaturity (MR). The decision making process converts the sampling

eview results (e) to confidence (F).The dependencies among the model components can be

xpressed by the above functions. For example, S (system) and (governmental regulations) affect the resource consumption atreparation (Rp) and review time (RR); these dependencies arehown through functions f1, f2, and t. In the above case stud-es, Chooz B project using a parallel Transputer system requiredxtraordinary efforts in preparation and review (Rp, RR). This is aase in which System (S) affects Resource consumption (Rp, RR). Onhe other hand, for the G (government) regulations, that USNRC reg-lations (SRP Chapter 7) are more strict than Japanese regulationsJAPG4609) may affect the amount of resources (Rp, RR) needed.

The relations among the above three profiles (PRE, PEC, PEF) cane expressed as

Rp · PRE

PEC· PEF = F.

ots refer to conversion. Fig. 2 gives a sample to explain theseodel components. Assume that a license applicant performs

he QA activities including requirement Analysis (A), Testing (T),eview (R), and Audit (Au) using equal resources; that is, each takes5% of the total resources. This is shown by the PRE distribution inig. 2a. Given a total resource amount, say, 100 K, then, the cost forach type of QA activities is 25 K. If the unit costs for the analysis,esting, review and audit evidence are 40 K, 20 K, 25 K, and 10 K,espectively, as shown in Fig. 2b, then the effective evidence forach type can be calculated; namely, their ratios are 25/40: 25/20:5/25: 25/10, as shown in Fig. 2c. Note that the amount of evidencebtained by the requirement A (analysis) is the lowest since it hashe highest unit cost. The effective evidence will then be convertednto reviewer’s confidence through PEF and be accumulated into

eviewer’s faith (F), as shown in Fig. 2d where “AC” refers to thecceptance level of faith.

This model provides a framework such that further investi-ations can be conducted by analyzing relations among model

Fig. 3. Different profile alignment.

components. These model components are mainly efficiency-related; however, the maturity level (MR and MA) and theevidence–confidence profile PEF also deal with regulatory effective-ness.

3.3. Conditions for optimal certification performance: alignedprofiles

From the model we can define the optimal certification per-formance as that achieving the maximal confidence under a fixedcertification resource. The optimal certification performance pro-cess is shown in Fig. 3. The confidence conversion process showedthat conditions that lead to the optimal certification performanceare closely related to the alignment relation among three evi-dence profiles, i.e., PRE, PEC and PEF. The exact optimal conditioncan be derived by mathematical techniques if the mathematicaldefinition of each profile is known. In general, the more the confi-dence generated by a certain type of evidence, the more resourcesshould be devoted to that type of evidence. However, there aresome special cases when evidence can only be generated by acertain unique technique, such as qualification testing; then theevidence–confidence Profile (PEF) can be designed to reflect sucha situation. Thus, in most cases, we may conclude that the key tooptimal certification efficiency is the proper alignment of the threeevidence profiles PRE, PEC and PEF. Fig. 3a shows a good alignmentcase and Fig. 3b shows a poor alignment case. In the good alignmentcase (Fig. 3a), higher percentages of resources should be invested inthe QA activities which generate the evidence (E1, E3, and E5) thatcan convert to higher confidence (in PEF). Thus, the accumulatedconfidence (F) is high. In the bad alignment case (Fig. 3b), moreresource is invested in those activities whose evidence, (E1, E3, andE5) yields confidence slowly. A good alignment case usually leadsto successful certification.

Besides, saturation phenomenon occurs in theEvidence–Confidence Profile P since confidence will reach a

EFsaturated point after which more evidence will not increase confi-dence. Thus, the accumulated confidence F also has the saturationphenomenon.

3 ering a

4

fiplao

4

dydospai

4

sprtFf

4

Cyca

4

pntnbFdpo

4

fratT(Tfid

4

Khp

84 S. Yih, C.-F. Fan / Nuclear Engine

. Application of certification process model

The developed certification process model represents a simpli-ed model, but it captures essential features of real certificationractice. The advantage of having a model is that we can perform

ogical analysis and correlate results with real cases. Thus we canpply this model to explain previous certification experiences andbtain insights into the characteristics of certification process.

.1. Diagnosis of previous digital I&C certification cases

In this section we apply the developed model to explain thoseigital I&C experiences reported above. Fig. 4 summarizes our anal-sis results in a visual form with the model components explicitlyenoted. At the top part of each case, square boxes show the spacef government regulation (G), system (S), and evidence space (E);quare boxes along the certification process indicate functions: thereparation function (f; i.e. f1 and f2), threaded review function (t),nd confidence conversion function (g); the output-shape boxesndicate resulting evidence.

.1.1. Chooz B N4/P20Chooz B’s I&C project overlooked the severity of evaluation

pace explosion problem introduced by the parallel system Tans-uters. The choice of Transputers as its computing platformesulted in a prohibitive workload for V&V task, i.e., the evalua-ion space is too huge to be evaluated, as shown by numerous E′ inig. 4a. Unable to conduct complete V&V explains why P20 projectailed.

.1.2. Chooz B N4/ControlblocN4/P20 was later replaced by an older product developed by

ontrolbloc. The long operating experience of this older productielded high confidence. Fig. 4b shows the smooth licensing pro-ess, denoted by f, g, t functions performed by the applicant (MA)nd the regulator (MR).

.1.3. Sizewell BThe major problem stems from the misalignment of evidence

rofiles (PRE, PEC and PEF). The extra IV&V performed by NE didot contribute commensurate confidence return due to the facthat Westinghouse (WH) had already performed an effective inter-al V&V. The evidence–confidence (PEF) return for NE IV&V effortecame very low due to the saturation effect. This is shown inig. 4c. It is the saturation situation when more QA activities in evi-ence preparation do not generate more confidence. The evidencereparation in this case is an overkill activity due to the overlookingf the diminishing return effect of the evidence–confidence profile.

.1.4. DarlingtonDarlington Digital Shutdown System Project’s problem came

rom the mismatch between the applicant’s evidence and theegulator’s evidence–confidence profiles. Low confidence waschieved at first. Regulatory authorities adopt Dr. Parnas’ posi-ion, which considered more rigid analysis as necessary evidence.he utility had to re-submit formal analysis information using SCRHeitmeyer, 2001) for review and finally got the operation license.his case shows that formal methods, such as SCR, yield high con-dence. That is, PEF (Evidence = formal method) is high. This case isepicted in Fig. 4d

.1.5. KK6/KK7

The major reason for KK6/7’s success lies in the fact that

K6/KK7 I&C project adopted well-developed design that vendorsave accumulated more than 10 years’ experience. The cost forreparing evidence (PRE, PEC) was low and knowledge (M) was

nd Design 242 (2012) 379– 388

abundant and shared among stakeholders. Thus all evidence pro-files (PRE, PEC and PEF) were well aligned, and this resulted in aneffective certification process. This is shown in Fig. 4d. That is, PEF(Evidence = operating experience) is high.

4.1.6. Oconee digital upgradeThe major reason for Oconee’s lengthy licensing process

(2002–2010) was due to the insufficiency of IEEE 603-based evi-dence.

4.2. Insight gained from evaluation of certification process model

After applying the proposed model to evaluate the previouscertification experiences, we can obtain lessons learned and thusimprove our understanding about the nature and limitations of cer-tification process. Some insights gained from the above analysis arelisted below:

4.2.1. Concept of evaluation space and its implicationsEvaluation space is defined by regulations and system design,

i.e., G × S. This represents the scope to be worked with duringcertification process. The larger and complex of the evaluationspace, the more difficult the certification will be. This is shownby N4/P20 case. This feature implies that we shall choose simpli-fied digital devices for safety systems in order to have a small andmanageable evaluation space. This explains why PLC-based designis more popular than microprocessor-based design in digital I&Csafety applications; a PLC-based design is much simpler than amicroprocessor-based design for the same functionality. A simpledesign is desirable for a safety-critical computing system. Simplic-ity may enhance safety and facilitate the certification process.

Oconee licensing experience shows the importance of themutual understanding in the regulations (G) between the appli-cant and the regulator. Furthermore, if the logical structure of theregulations (S) can be explicitly represented in an organized way,it enhances the review efficiency and effectiveness because of theclear definition and common structures of the review evaluationspace for both the applicant and the reviewer.

4.2.2. Diminishing return behavior of evidence cost profileThe evidence–confidence profile (PFE, PRE, PEC) in general has a

diminishing characteristic. At beginning, the confidence level canincrease as more evidence is collected. But the tendency of incre-ment will saturate, i.e., up to a certain point, the confidence levelgained will diminish even more evidence is collected. Thus, beyondthe saturation point, invested resource will be wasted, as that hap-pened in Sizewell B PPS IV&V project. This feature can also be usedas an argument for reducing some controversial regulations, suchas independent requirements for V&V and safety analysis (USNRC,2007), as well as overwhelming documentation requirements, etc.

4.2.3. The impacts of human factors in certification processThis influence of human factors can be observed by examining

the proposed model. For functions f1, t, g, in our model, each processinvolves the proficiency maturity factor M (MA or MR) which is highlypeople-dependent and also can be highly subjective. Therefore, theperformance of software certification process is heavily affected byhuman capability. This is quite different from hardware-orientedequipment certification process. The ultimate goal of managinghuman factor for a certification process is to maintain consensusamong all stakeholders. In a broader sense, consensus represents

not only having similar interpretation of regulations but also hav-ing the same perception of evidence–confidence profiles. Withoutconsensus the certification process can be difficult and frustrating,as that happened in Sizewell B, Darlington, or Oconee projects. On

S. Yih, C.-F. Fan / Nuclear Engineering and Design 242 (2012) 379– 388 385

N4/P20Edf/France

N4/ControlblocEdF/France

DarlingtonOntario/Canada

f f

g

t

Cancelled

….….

E’

e

G S E

MA

G S E

E’ E’ E’

E’ E’

E’ E’

Confidence

KK6/KK7TEPCO/Japan

f

t

g

G S E

E’

e

Confidence

formal

F F

f

gPEF

PRE

G S E

E’

Confidence

PEF

F

F

…

f

g

e

SCR

…

Sizewell BNE/British

f

f’

Confidence

G S E

E’

g

t

e

E’

WHIV&V

NEIV&V

F

MA

MA

MR

(a) (e)(d)(c)(b)

PEC

transputer experienced

…

OconeeDuke Energy/USA

f

g

G S E

E’

F

(f)

Look forIEEE 603evidence

…

withdraw

f

g

E’

2nd time

F

603

….

MA MA

MAMR

MR

MR

MR

MR

MA

MR

Confidence

PEF

Olderproduct

l I&C c

tfiK

4y

Nsmrmpib

4m

uamcms(ut

management (CRM)Conventional compliance-based regulation is based on the

assumption that the behavior of the activities or equipment under

Fig. 4. Diagnosis of digita

he other hand, when all people reach consensus, then the certi-cation process can be smooth and efficient as that happened inK6/7 project.

.2.4. Evidence using formal methods and operational experienceields high confidence

As shown in the case studies, operational experiences in Chooz B4/Controlbloc and the KK6/7 generate the regulator’s confidence,

o did the formal method SCR used in Darlington’s case. Formalethods provide a high Evidence–conFidence (PEF) ratio, but also

equire a high Evidence–Cost ratio (PEC). Operational experienceay not be available in all cases; thus, using formal methods to

repare certification evidence seems to be a promising approach toncrease regulator’s confidence. Thus, formal methods can improveoth regulatory review efficiency and effectiveness.

.2.5. The essence of managing certification process: resourceanagement under uncertain environment

The key words appeared in SRP (USNRC, 2007), the review reg-lations for nuclear software, include subjective judgment suchs: “reasonable assurance” and “sufficient”. All of these key require-ents depend on personal judgment. The subjectivity invokes

ertain degree of flexibility and unpredictability for the perfor-ance of certification process. In the model we represent such

ubjectivity judgment in the form of evidence–confidence profilePEF) and maturity proficiency (M). In reality, there is significantncertainty associated with these evidence profiles. The uncer-ainty comes from both stochastic nature (aleatory uncertainty)

ertification experiences.

and lack of complete knowledge (epistemic uncertainty) aboutthe behavior of evidence–confidence relation. Without accurateevidence–confidence and evidence–cost information, the man-aging of certification process becomes a resource managementprocess under highly uncertain environment.

4.2.6. Principle of effective certification process: continuous risk

Fig. 5. The relations of main items.

386 S. Yih, C.-F. Fan / Nuclear Engineering and Design 242 (2012) 379– 388

Threats (4)

Functional requirements

FAF_AUX: Auxiliary features (5.12)

TDB_EXT: External Threat

ACO_CPA: Completion of protectiveAction (5.2)

FHF_CON: Control Access (5.9)

FSD_RES: Restriction on sharingbetween units

FHF_HUM: Human Factor (5.14)

FSD_DID: Defense-in-Depth

FSD_RED: Redundancy (5.6)

FSD_NVE: N-version

TND_IAC: Illegal Access and Control

FOB_CON: Bypass Condition (6.6)FOB_ALG: Bypass algorithm (6.6)

FID_MCA: Display for ManuallyControlled Actions (5.8.1)

FID_SSI: System Status Indication (5.8.2)

FID_IOB: Indication of Bypass (5.8.3)

FID_LOC: Location (5.8.4)

FRE_REP: Repair (5.10)

FSC_ATC: Automatic Control (6.1)

FSC_MAC: Manual Control (6.2)

FTC_TAC: Test and Calibration (5.7,6.5)

Assessment

Performance

Structural requirements

ARE_GOA: Reliability Goals (5.15)

AQU_QUA: Quality (5.3)

AQU_EQU: Equipment qualification (5.4)

FIN_RED: Independence betweenredundant portions of a safety system (5.6.1)

FIN_CPI: Between safety system andDBE(5.6.2)

FIN_OTH: Between safety systemsand other systems (5.6.3)

IEEE Std 603 &7-4.3.4

AID_IDE: Identification (5.11)

ASF_SGF: Single failure criteria (5.1)

Assurance requirement

Critical Asset Constraints

Defensive measures

TDB_INT: Internal Threat

CDB_CON: Constraints (4, 6.8)

TND_OBE: Operational Bypass Error

TND_STF: Structural Failure

ACO_SIN: System Integrity (5.5)

es and

rbsbrcet(iHct

5

tdulltcod

Fig. 6. Proposed safety class

egulation is deterministic and predictable; confidence can, thus,e established thereafter. For digital I&C equipment and its relatedoftware development activities, their performance are neither sta-le nor predictable for most of the time. Thus, compliance-basedegulation approach often results in resource waste due to the inac-urate estimation of the evidence–confidence profile (PEF) and thevidence–cost profile (PEC). In principle, such an inherent uncer-ainty problem can be alleviated by Continuous Risk ManagementCRM) technique, which manages risks in a project throughoutts lifetime. Accurate evidence profiles assessment is still difficult.owever, the mismatched gap between various evidence profilesan be narrowed by continuously assessing and mitigating risk ashe project goes on. CRM also enhances regulatory effectiveness.

. A prototype of a computer-aided licensing system

As indicated above, human factors play important roles in cer-ification and an explicit logical structure of the regulations isesirable. Thus, a computer-aided licensing tool based on a reg-lation ontology may alleviate the problems and streamline the

icensing review process. Current licensing process still involves aot of clerical work, for example, document preparation, and syntac-

ic evidence gathering, which may be automatically performed byomputer-aided tools. Therefore, we have constructed a prototypef a computer-aided licensing system based the proposed model toemonstrate how to further enhance regulatory review efficiency.

families based on IEEE 603.

First, explicit logical structure of the regulations has to bedefined. We took IEEE 603 (1998) for certification of safety sys-tems in nuclear domain as our case study. IEEE 603 is writtenin natural language and lacks clear logical structures. To speedupevidence preparation and review process, we propose the followingsteps for the tool construction:

1. Reorganize and construct the logical structures of IEEE 603requirements.

2. Represent the identified requirement by XML (W3C, 2010) tags.3. Construct associated review tools using the above tags.

The main concepts in IEEE 603 are proposed to be classified intothreats, critical asset constraints, defensive measures, and assurancerequirements. Fig. 5 depicts the relationships among these proposedstructures. Threats may cause system critical assets exceedingsafety ranges, and thus lead the system to an unsafe state. Defen-sive measures would prevent this from happening. However, whenthese measures may have flaws, then critical asset constraintsshould be the next layer of protection; if this does not work, thenhazardous events may occur. The assurance requirements are usedto ensure the quality of the defensive measures and the critical asset

constraints.

We then use Common-Criteria-like (ISO/IEC, 2006, 2007a,2007b) component representation to represent IEEE 603 require-ments. Common Criteria (CC) or ISO/IEC 15408 (2006, 2007a,

S. Yih, C.-F. Fan / Nuclear Engineering and Design 242 (2012) 379– 388 387

mark

2tacop

uAsgct

Fig. 7. A

007b) is an international standard for computer security cer-ification. It provides a common set of security functional andssurance requirements for certification in a highly organizedlass–family–component hierarchy. Fig. 6 shows the first few levelsf the IEEE 603 requirements in a CC fashion in which numbers inarentheses indicate section numbers in the standard.

These identified requirements of the regulations can then besed as XML tags (W3C, 2010) to mark up the submitted documents.

prototype of a computer-aided licensing system is then con-

tructed. The prototype provides a mark up tool and a review tableeneration tool. The mark up tool shown in Fig. 7 assists the appli-ant to tag his submittals using the above identified tags. Assumehat a list of review questions is prepared and the associated tags

Fig. 8. Generated r

ing tool.

for each question have been determined before hand. Then, thereview-table-generation tool can automatically extract related evi-dence from the tagged submittals, as shown in Fig. 8 to reducesyntactic evidence search effort for the regulator. Moreover, evi-dence can be classified into several types, say, claim, testing, audit,operation experience, and formal methods with different confor-mance levels (fully, largely, partially, and none). These are shownin Fig. 8. Furthermore, the three evidence profiles (PRE, PEC, PEF) canalso be set and shown by the tool. The percentage of resource uti-

lization PRE can be provided by the applicant; the regulator may keyin his confidence conversion profile PEF. These profiles and relatedanalysis help to examine review efficiency and to enhance the pre-dictability of the decision making process. This licensing support

eview form.

3 ering a

srrd

6

amwdwciitafadtftftitiBmFceasy

R

A

C

C

D

D

F

F

F

G

G

Chin-Feng Fan received the Ph.D. degree in computer science and engineering from

88 S. Yih, C.-F. Fan / Nuclear Engine

ystem may have the following advantages: (1) provide a commonegulatory structure for both the applicant and the regulator, (2)educe clerical work, and (3) improve the transparency and pre-ictability of the licensing process.

. Conclusion

In this paper we developed a Regulatory Decision Making Modelnd reported our survey which applied this model to evaluation ofajor digital I&C certification experiences. The motivation is thate noticed that there exist performance variances among differentigital I&C certification cases, i.e., some were smooth and efficienthile others were lengthy and problematic. In order to explain the

auses of such variances and to develop techniques for improv-ng the efficiency of certification process, we have conducted basicnvestigation in regulatory decision making. By viewing the cer-ification process as an evidence confidence conversion process,

Regulatory Decision Making Model is developed and used as aramework to analyze the behavior characteristics of certificationctivities. The model identified major factors and functions thatominate the performance of certification process, among whichhe proficiency maturity and evidence profiles are the most criticalactors affecting the certification process. We were able to applyhis model to explain why some certification cases are success-ul and some are troublesome. The successful application showshe validity of the proposed model. We presented several insightsnto the nature of certification process. Our evaluation identifiedhat inherent uncertainty associated with the various profiles dom-nates and limits the potential performance of certification process.ased on our study we consider that Continuous Risk Manage-ent technique should be applied to cope with such uncertainty.

urthermore, we took IEEE 603 as a case study to show how toonstruct computer-aided tools based on the derived insights tonhance the efficiency of a regulatory review. The proposed modelnd approach are general enough and can be applied to otherafety-critical computing domains for regulatory efficiency anal-sis.

eferences

ppell, B., 1992. Putting in a replacement for Controbloc P20 at Chooz B. In: NuclearEngineering International, July, pp. 45–48.

hun, C., et al., 2000. Regulatory assessment of the Darlington shutdown system tripcomputer software redesign. In: Proceedings of International topical meeting onnuclear power plant instrumentation, control and human–machine interfacetechnologies, NPIC&HMIT 2000, Washington, DC, USA.

raigen, D., et al., 1994. Case study: Darlington Nuclear Generating Station. IEEESoftware (January), 30–32, 1994.

epartment of Energy, 2005. DOE NP2010 nuclear power plant construc-tion infrastructure assessment. http://www.ne.doe.gov/np2010/reports/mpr2776Rev0102105.pdf.

MID sample regulatory file review tool. Available from: http://www.niaid.nih.gov/LabsAndResources/resources/DMIDClinRsrch/Documents/rfrt.doc (accessedNovember 2010).

enton, N.E., et al., 1998a. Assessing dependability of safety critical systems usingdiverse evidence. IEE Proceedings Software Engineering 145 (1), 35–39.

enton, N.E., 1998b. A Strategy for Improving Safety Related Software EngineeringStandards. IEEE Transactions on Software Engineering 24 (11), 1002–1013.

ukumoto, A., et al., 1998. A verification and validation method and its applicationto digital safety systems in ABWR nuclear power plants. Nuclear Engineeringand Design 183 (2), 117–132.

angemi, A., et al., 2003. Some ontological tools to support legal regulatory compli-ance with a case study. In: Proceedings of Workshop on Regulatory Ontologies,OTM’03, pp. 607–620.

ran, B.A., Dahll, G., 2000a. Estimating dependability of programmable systemsusing Bayesian belief nets. OECD HALDEN Reactor Project Report (HWR-627).

nd Design 242 (2012) 379– 388

Gran, B.A., et al.,2000b. Estimating dependability of programmable systems usingBBNs. In: Safecomp 2000 (LNCS 1943). Springer, pp. 309–320.

Gran, B.A., 2001. Applying Bayesian belief net in software safety assessmenton a real, safety related programmable system. In: Zio, E., et al. (Eds.),Safety & Reliability, Towards A Safer World. Politecnico di Torino, Torino, pp.1045–1052.

Guesnier, G., et al., 1989. C&I systems for France’s N4 NPPs. In: Nuclear EuropeSeptember–October 1989, pp. 17–18.

Heitmeyer, C., 2001. Software cost reduction. In: Marciniak, J.J. (Ed.), Encyclopediaof Software Engineering. Wiley-Interscience, p. 2001.

Hung, E., 2008. Darlington digital control computer system replacement approachand experience. In: IAEA Technical Meeting on Impact of Digital I&CTechnologies on the Operation and Licensing of NPPs, Beijing, China,November 2008.

IEEE 603, 1998. IEEE standard criteria for safety systems for nuclear power generat-ing stations.

ISO/IEC 15408, 2006. Common Criteria for information technology security evalua-tion. Part 1. Introduction and general model, Version 3.1. September, 2006.

ISO/IEC 15408, 2007a. Common Criteria for information technology security evalu-ation. Part 2. Security functional components, Version 3.1. September, 2007.

ISO/IEC 15408, 2007b. Common Criteria for information technology security evalu-ation. Part 3. Security assurance components, Version 3.1. September, 2007.

Japan Electric Association, JEAG 4609-1999, 1999. Application criteria for pro-grammable digital computer system in safety-related system of nuclear powerplants.

Jee, E., et al., 2010. Automated test coverage measurement for reactor protectionsystem software implemented in function block diagram. In: SAFECOMP 2010,Vienna, Austria.

Kang, H., et al., 2009. Reliability assessment of a safety-critical software by usinggeneralized Bayesian nets. In: Sixth American Nuclear Society International Top-ical Meeting on Nuclear Plant Instrumentation, Control, and Human–MachineInterface Technologies, Knoxville, Tennessee.

MacLachlan, A., 1994a. I&C Woes behind It, EDF on target for startup of first N4reactor. Nucleonics Week (July), 3–4.

MacLachlan, A., 1994b. French regulators ‘Lost Hope’ of proving Chooz-B digital I&Csystem. Inside NRC (May), 6–7.

Marshall, P., Silver, R., 1993. Sizewell B computer controversy looms over fuel loadschedule. Nucleonics Week 34 (October (42)).

Neil, M., et al., 2008. Modelling dependable systems using hybrid Bayesian networks.Reliability Engineering and System Safety 93 (7), 933–939.

Nuclear Energy Agency (NEA), 2009. Improving nuclear regulation: compilation ofNEA regulatory guidance booklets.

Nuclear Engineering International, 1993. Sizewell B reactor protection reliability:nuclear electric presents its case. Nuclear Engineering International (March),28–33.

Peyrouton, M., Pirus, M., 1993. Progress on N4 I&C architecture. In: Proceedings ofTopical Meeting on Nuclear Plant Instrumentation, Control and Man–MachineInterface Technologies, Oak Tenn., USA, 1993, pp. 305–311.

Seong, P.H. (Ed.), 2009. Reliability and Risk Issues in Large Scale Safety-Critical DigitalControl Systems. Springer.

Sivertsen, T., 1996. A case study on the formal development of a reactor safetysystem. In: FME ‘96, Industrial Benefit and Advances in Formal Methods, pp.18–38.

Sparrow, M.K., 2000. The Regulatory Craft: Controlling Risks, Solving Problems, andManaging Compliance. Bookings Institution Press, Washington, DC.

Spring, N., 2010. NRC approves first digital I&C system. Power-Gen worldwide mag-azine. Available from: http://www.powergenworldwide.com/index/display/articledisplay/0478279771/articles/powergenworldwide/nuclear/equipment/2010/02/NRC-approves-first-digital-I-and-C-system.html (accessed September2010).

USNRC, 2007. NUREG-0800, Chapter 7, Instrumentation and Controls, StandardReview Plan (SRP) for the review of safety analysis reports for nuclear powerplants.

W3C, 2010. Extensible Markup Language (XML) 1.0. www.w3.org.

Swu Yih received the Master degree in nuclear engineering from Iowa State Uni-versity, Iowa, USA, and the Ph.D. degree in computer science and engineering fromSouthern Methodist University, Dallas, USA. He had worked for Institute of NuclearEnergy Research, Taiwan, for more than 30 years, before switching to academia. Heis currently an associate professor at Computer Engineering and Information Sci-ence Department of Ching Yun University, Taiwan. His research interests includesoftware safety, software reliability, and nuclear regulation.

Southern Methodist University, Dallas, USA. She is currently an associate professorat Computer Science and Engineering Department of Yuan-Ze University, Taiwan.Her research interests include software engineering, software safety analysis, andregulatory compliance review.