at SciVerse ScienceDirect

Progress in Nuclear Energy 55 (2012) 12e32

Contents lists available

Progress in Nuclear Energy

journal homepage: www.elsevier .com/locate/pnucene

A verification exercise in multiphysics simulations for coupled reactor physicscalculations

Vijay S. Mahadevan a, Jean C. Ragusa b,*, Vincent A. Mousseau c

aNuclear Energy Division, Argonne National Laboratory, Argonne, IL 60439, USAbDepartment of Nuclear Engineering, Texas A&M University, College Station, TX 77843, USAcMultiphysics Simulation Technologies, Sandia National Laboratories, Albuquerque, NM 87185, USA

a r t i c l e i n f o

Article history:Received 16 November 2011

Keywords:Multiphysics couplingCode verificationMethod of manufactured solutionsConvergence order

* Corresponding author.E-mail addresses: mahadevan@anl.gov (V.S. Maha

(J.C. Ragusa), vamouss@sandia.gov (V.A. Mousseau).

0149-1970/$ e see front matter � 2011 Elsevier Ltd.doi:10.1016/j.pnucene.2011.10.013

a b s t r a c t

The modeling of nuclear reactors involves the solution of a multiphysics problem with various time andlength scales. Mathematically, this requires solving a system of coupled, nonlinear, stiff, Partial Differ-ential Equations (PDEs). This paper deals with the verification aspects associated with a multiphysicscode, i.e., the substantiation that the mathematical description of the multiphysics equations are solvedcorrectly (in time and space). Multiphysics applications have the added complexity that the solution fieldparticipates in various physics components, potentially yielding spatial and/or temporal coupling errors.We present a multiphysics framework that tightly couples the various physical models using theJacobian-free Newton-Krylov technique (JFNK) and show that high-order convergence can be achieved inboth space and time. Code verification results are provided.

� 2011 Elsevier Ltd. All rights reserved.

1. Introduction

In reactor applications, multiphysics simulations combinemodels and algorithms from a diverse set of disciplines that need tobe consistently coupled to yield accurate answers. Traditionalcoupling methods (also referred to as operator-split (OS) couplingtechniques) lag nonlinearities, resulting in a loss of accuracy andstability (Strang, 1968; Marchuk, 1971; Mousseau et al., 2000).Recently, researchers have considered a different approachwhereby the whole nonlinear multiphysics is solved at once,avoiding the lag in the coupled physics components (Knoll andKeyes, 2004; Knoll et al., 2003).

The present paper is concerned exclusively with the verificationof numericalmethods formultiphysics applications through the useof manufactured solutions. It is an exercise in mathematics, whereone ensures that the equations are solved correctly, i.e., that thesoftware has been coded and implemented correctly. This is anintegral part of the development process for all modeling softwarealongwith validation and uncertainty quantification. The process ofcode verification usually involves a combination of unit tests andwhole system tests.We do not discuss unit testing in this paper (oneshould always perform unit testing), but rather focus on solutionverification efforts applied to multiphysics problems as a whole.

devan), ragusa@ne.tamu.edu

All rights reserved.

The process of code verification usually includes a combinationof the following techniques: (1) comparison against analyticalsolutions (when such solutions are available; this often requiressome drastic model and geometrical simplifications), (2) conver-gence order estimation using various space/time grid refinements,and, (3) convergence order estimation using the Method of Man-ufactured Solutions (MMS) (Roache, September, 2009) (this oftenrequires that intricate volumetric and boundary source terms becomputed accurately). In the context of nonlinear multiphysicsapplications, these same recipes apply with the added difficultythat the manner inwhich the physics components are coupled maydegrade the convergence order in time (e.g., when using anoperator-split technique) and/or the convergence order in space(e.g., when projections and interpolations of field variablesbetween source and target meshes for different physics compo-nents are required).

Firstly, in order to eliminate the temporal pollution error due tooperator-splitting, fully implicit numerical methods for tightcoupling of various physics have been proposed for multiphysicsapplications (Knoll et al., 2003; Mousseau, 2004) and are employedin this paper. These techniques, based on Newton’s method appliedto the whole nonlinear problem, are briefly described in Section 3.With a fully implicit discretization, the temporal coupling error canbe controlled and reduced to a small enough user-specified level,hence allowing for a proper verification of the temporal conver-gence orders. Secondly, each physics component may be dis-cretized on its own mesh (the source mesh) and the solution field

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 13

from a given physics needs to be exported onto another mesh (thetarget mesh). L2 projection or interpolation of the solution betweenthe source and targetmeshesmay cause non negligible spatial error(de Boer et al., 2007). In order to minimize the spatial couplingerror due to the data transfer between the different physics definedon possibly non-overlapping meshes, several techniques have beendeveloped (Grandy, 1999; Jiao and Heath, 2004). Jiao and Heathhave derived rigorous cost estimates for different remappingmethods along with the corresponding solution computationaltimes (Jiao and Heath, 2004). The spatial coupling error due to, forinstance, the use of different meshes, is still an ongoing topic ofresearch (Tautges, 2004). A detailed study of this effect is beyondthe scope of this paper and wewill mostly use identical meshes forall physics components throughout the paper, with the exception ofone numerical test case for which different spatial meshes areemployed and convergence rates are measured. When differentmeshes are employed here, the spatial coupling error betweenphysics components is controlled using an appropriate numericalquadrature rule, see Section 3.1.2.

The outline of the paper is as follows: in Section 2, we presentthe physical models employed in this study. In Section 3, we reviewnumerical methods for fully implicit tightly coupled simulationsbased on the Jacobian-free Newton-Krylov technique, as well as thetemporal and spatial discretization techniques. In Section 4, wepresent a test-bed software framework developed to performmultiphysics simulations for reactor applications. In Section 5, wereview the techniques used for code verification. Numerical resultsare given in Section 6 and conclusions and recommendations areproposed in Section 7.

2. Physical models

In order to demonstrate the feasibility of high-order space/timemethods for multiphysics applications in reactor analysis and topresent the approach utilized to verify code implementation, thefollowing physical models are employed. These models have beendeliberately chosen to be “coarse-grained” because the purpose ofthe paper is not to validate a model but to present a multiphysicsverification study. However, in Section 4, we describe how higher-fidelity physical models can be interchanged easily within the JFNKframework, when suchmodels are deemed necessary. In this paper,we employ the following representations pertaining to the physicsinvolved in reactor applications: 3D multigroup neutron diffusion,3D nonlinear heat conduction, and (multiple) 1D channels forsingle-phase fluid flow.

2.1. Multigroup neutron diffusion model

The time-dependent neutron multigroup flux is obtained bysolving the following equations

1vg

vfgðr; tÞvt

� V/$Dgðr; tÞV

/fgðr; tÞ þ St;gðr; tÞfgðr; tÞ

¼XGg0 ¼1

Sg0/gs ðr; tÞfg0 ðr; tÞ þ cgp

XGg0 ¼1

�1� bg

0�nSg0

f ðr; tÞfg0 ðr; tÞ

þXJj¼1

cgd;jljCjðr; tÞ cg˛½1;G�; (1)

whereG is the total number of energygroups, J is the total number ofdelayed neutrons groups. The notation used is standard in text-books. The system of equations is closedwith appropriate boundaryand initial conditions. The neutronic model employs fuel assembly

homogenized cross sections. Cross sections are either tabulated orprovided in a closed form approximation, as a function of fueltemperature and coolant density (extension to additional parame-ters, such a boron concentration, void history, control rod history,etc., is straightforward). The tabulated cross-section values are ob-tained using table look-up and Rp interpolation, where p is the totalnumber of parameters used. The energy production due to fissionand radiative capture events is given by

q000ðr; tÞ ¼XGg¼1

hkgf S

gf þ kgcS

gc

iðr; tÞfgðr; tÞ; (2)

where the k coefficients represent the amount of energy releasedper reaction event. The ODEs for the evolution of delayed neutronprecursor concentrations are given by

dCjðr; tÞdt

þ ljCjðr; tÞ ¼ bjXGg0 ¼1

nSg0f ðr; tÞf

g0 ðr; tÞ: (3)

2.2. Thermal heat conduction model

The nonlinear heat conductionmodel employed here representsthe core as a porous medium and is described by the following PDE

rCpvTvt

� V/$kðTÞV

/T ¼ q000ðr; tÞ; (4)

with appropriate boundary and initial conditions. q0 0 0

is thevolumetric heat source in the fuel; the density r, the specific heatCp, and the conductivity k may depend on the temperature T. Inthe current model, the heat conduction equation is solved on thesame domain as the neutronic model. In the Results section, weprovide verified solutions for cases with identical and separatespatial meshes for a coupled neutronics/heat conductionproblem.

The fuel conduction physics can be coupled to the 1-D fluid flowphysics in which conjugate heat transfer occurs at the fuelefluidinterface. The boundary term coupling the conducting solid to thefluid is given by

�kðTÞvnT jw ¼ hc�Tw; Tf

��Tw � Tf

�; (5)

where Tw is the (solid) wall temperature, Tf is the coolant temper-ature, and hc(Tw, Tf) is the convective heat transfer coefficient.

2.3. Coolant flow model

The coolant is modeled using a single-phase fluid flowingvertically in 1-D channels. Themodel allows for one ormultiple 1-Dchannels (the maximum number of channels being equal to thenumber of fuel assemblies; a simple user-defined mapping isemployed to assign channels and fuel assemblies). The governingequations for the fluid flow are solved in terms of the conservativevariables and are as follows

vtrþ vzðruÞ ¼ 0 (6)

vtðruÞ þ vz

�ru2�þ vzP ¼ � fw

Dhyrjuju (7)

vtðrEÞ þ vzðuðrE þ PÞÞ ¼ S; (8)

where r is the fluid density, ru its momentum, rE its total energy, Pthe pressure, and fw/Dhy the wall-friction factor divided by the

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3214

hydraulic diameter. S is the external source terms representing theenergy convected from the fuel and is given by

S ¼ hc�Tw; Tf

� �Tw � Tf

� Pwet

Ahy(9)

where Pwet/Ahy is the ratio of the wetted perimeter to the hydraulicarea of the flow. Dhy, Pwet, and Ahy are user-supplied values. Anequation of state closes the system of equations

P ¼ f EoSðr; reÞ; (10)

where the internal energy is re ¼ rE�1/2ru2.

3. Numerical discretization

The above physical models yield a system of nonlinearly coupledequations of the form

vuvt

¼ fðuÞ; (11)

where u is the solution vector, f is a nonlinear vector function,f : RN/RN , and N is the total number of unknowns. It helps torepresent u as a vector comprised of the solution vector for each ofthe M physics components involved, i.e., u ¼ [u1, u2,., uM]T. Asimilar definition holds for the nonlinear residual f(u) where itsmth component is the nonlinear residual stemming from the m -thphysics component. Note that fm(u) may depend effectively on allother fields, e.g., fm(u) ¼ fm(u1, u2, ., uM). In this paper, all thephysical models are represented in the form of Eq. (11) forsimplicity.

T

C A

B,

3.1. Spatial discretization

3.1.1. Continuous and discontinuous finite element discretizationsThe spatial discretization is performed using the Finite Element

library libmesh (Kirk et al., 2006). A continuous Galerkin (cG)method (Solin et al., 2003) with Lagrange basis functions is utilizedfor elliptic/parabolic PDEs (e.g., neutronics and heat conductionmodels), whereas a discontinuous Galerkin (dG) method(Hesthaven and Warburton, 2008) with Legendre basis functions isemployed for hyperbolic systems (fluid flow equations). Awealth ofdocumentation is available in the literature regarding cG and dGmethods and we refer the reader to (Solin et al., 2003; Hesthavenand Warburton, 2008; Zienkiewicz et al., 2005) and the refer-ences therein for additional details. The weak forms for the equa-tion systems are given in Appendix A.

3.1.2. Spatial coupling errors in multi-mesh approachesHere, we provide some details regarding how the spatial dis-

cretization approach handles the source and target meshes whenthese are different. Data exchange may be complex to performaccurately and care must be taken to ensure that important quan-tities are conserved in that process, see, for instance, the discus-sions in (Grandy, 1999; Jiao and Heath, 2004; Timothy and Caceres,2009). Recently, an approach based on adaptive mesh refinementtechnology has been proposed to handle multi-meshes simulations(Solin et al., 2010). As discussed in (Jiao and Heath, 2004), weemploy high-order quadrature rules for the numerical integrationof the terms residing on the target mesh such that the mass matrixand the load vectors are integrated exactly (Fix, 1972) in order toretain a conservative discretization and consistency in L2. Thisstrategy is generic and can be applied for arbitrary meshes,provided that the solution for a given physics and the propertiescan be evaluated at any given point in the target mesh. Hence, the

multi-mesh errors are reduced as the order of the numericalquadrature is increased.

For illustration, let us consider two physics components,indexed by 1 and 2. In the weak formulation, the nonlinear residualof physics 1, f1(u1, u2), is multiplied by a test function, bj

1. Thefollowing integral needs to be computed accurately for every cell K1of physics 1:ZK1

f1ðu1ðxÞ;u2ðxÞÞbj1ðxÞdx: (12)

Expanding the solution fields onto the basis functions, u1ðxÞ ¼Pi b

i1ðxÞui

1 and u2ðxÞ ¼ Pi b

i2ðxÞui

2, and replacing the integral bya numerical quadrature (wq, xq) yields

Xq

wqf1

Xi

bi1�xq�ui1;Xi

bi2�xq�ui2

!bj1

�xq�: (13)

For identical meshes, bi1ðxqÞ ¼ bi

2ðxqÞ and these quantities aresimple to obtain: mapping K1 onto a reference element is advanta-geous since the basis functions need only to be evaluated once at thequadrature points of the reference element. However, when themeshes are different, (i) the numerical integration needs to becarried out on the physical elementK1, and, (ii) all the cells of physics2overlappingK1need tobe retrievedand thebasis functionsb

i2 need

to be evaluated at the quadrature points, (xq). For general unstruc-tured meshes, one cannot obtain straightforwardly bi

2ðxqÞ in thereference element since this involves reverse lookups to find thecorrect target element for physics 2 containing the physical point.Hence, the numerical integration over a cell is carried out on the realgeometry (theactual element itself) andnoton itsmapped referenceelement. High order quadrature rules for each physics are employedalong with inverse mapping of the meshes in different physics inorder to evaluate the basis functions at the givenphysical points. Thedata structure for this mapping is part of the libmesh library. Thiscomputation is necessary each time the residual for a given physicscomponent needs to be evaluated and an efficient linked list datastructure is created to store the required information inmemoryandspeed up the integration over the cells.

3.2. Time discretization

The ability of achieve high order temporal discretization ofcoupled physics simulation depends on themethodology employedto tackle the nonlinear system of equations arising from thedifferent physics components. Because of the great disparity of timescales involved, implicit time discretization is preferred for stabilityreasons. We have chosen time integrators based on the DiagonallyImplicit Runge-Kutta (DIRK) family of methods. After spatial dis-cretization, a large system of coupled nonlinear Ordinary Differ-ential Equations (ODEs) is obtained

MdUdt

¼ fðUÞ: (14)

M is the mass matrix resulting from the spatial discretization(the use of a finite volume technique in space or a lumpednumerical spatial quadrature results in M being diagonal). In orderto describe the DIRKmethods used to discretize Eq. (14) in time, weintroduce the Butcher tableaux (Hairer and Wanner, 1996)

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 15

with B ¼ [b1, ., bs]T, C ¼ [c1, ., cs]T, and A ¼ ðaijÞi;j¼1;.;s. An s-stage DIRK method can be expressed as

Unþ1 ¼ Un þ DtXsi¼1

biki; (15)

where vectors ki (i ¼ 1,., s) are obtained by solving the following snonlinear systems

Mki ¼ f

0@tn þ Dtci;U

n þ DtXsj¼1

ai;jkj

1A: (16)

Performing the following change of variables

Zi ¼ Un þ DtXsj¼1

ai;jkj (17)

and substituting Eq. (17) in Eq. (16) yields a new set of snonlinear problems of the form

FðZÞ ¼ MUn þ DtðA5IÞfðtn þ DtC;ZÞ �MZ ¼ 0: (18)

Eq. (18) represents the nonlinear system of equations obtainedafter space and time discretizations of the whole multiphysicsproblem under consideration. Methods based on either Picard orNewton iterations can be employed to solve Eq. (18) and aredescribed in the next paragraph. The following temporal schemeshave been chosen: Backward Euler (BE), a second-order singly DIRKwith two stages (SDIRK2(2)), a third-order singly DIRK with threestages (SDIRK3(3)) and a fourth-order singly DIRK with five stages(SDIRK5(4)). All temporal schemes are A-, L-stable (non physicaloscillations are damped for large time step sizes) and stiffly accu-rate (non-degradation of convergence order for stiff problems); formore details on these time integrators, we refer the reader to(Hairer and Wanner, 1996; Norsett, 1986). The Butcher tableaux forthe schemes of interest are given below.

1 1

1TB

BE1(1)

3.3. Nonlinear solvers

The system of nonlinear equations, Eq. (18), is solved using OSand JFNK techniques. These are discussed next.

3.3.1. Operator-SplittingTraditionally, the design and analysis of nuclear reactors is based

on a solution procedure involving several mono-disciplinaryphysics codes, coupled together in a loose manner (Ober andShadid, 2004; Ropp et al., 2004; Mahadevan, 2006; Ragusa andMahadevan, 2009), a technique mathematically referred to asOperator-Splitting (OS). OS is still the most commonly employedcoupling strategy in the reactor physics community. Data exchangebetween physics components is often carried out using parallelvirtual machines (PVM) and message passing interfaces (MPI). Inthis coupling paradigm, the output data of one code is simplypassed on as the input data to another code at each time step. Often,this strategy is non-iterative and the nonlinearities due to thecoupling are not resolved over a time step, reducing the overallaccuracy in the time stepping procedure to first-order in time,although high-order time integration might have been used for theindividual physics components (Mahadevan, 2006; Ragusa andMahadevan, 2009). Even though more accurate OS strategies exist(Strang,1968;Marchuk,1971). they are not inwidespread use in thereactor analysis community.

3.3.2. Jacobian-free Newton Krylov (JFNK)Brown and Saad proposed (Brown and Saad, 1990) a matrix-free

version of Newton’s method that alleviates the explicit storage ofa Jacobian matrix in memory. By tackling the whole nonlinear asa system of equations that is amenable for the application ofNewton’s method, tight coupling between the various physicscomponents can be preserved and the use of high-order methodsfor time integration becomes viable. At each Newton iteration,a linear system of equations involving the Jacobian matrix needs tobe solved. As the number of physics components grows, so does thenumber of unknowns, resulting in possibly a largememory usage tostore the Jacobian matrix. Furthermore, assembling this matrixwhen many physics components are involved is not a trivial task.However, employing a Jacobian-free approximation avoids theneed for the Jacobian matrix storage since only the action of theJacobian matrix on a vector is required to solve the linear system.The resulting Jacobian-free Newton Krylov (JFNK) method hasenjoyed much success in recent years in several multiphysicsapplications; see the review articles (Knoll and Keyes, 2004; Gastonet al., 2009).

Eq. (18) describes the discretized multiphysics problem thatneeds to be solved for each of the s stages of the chosen DIRKmethod. Applying Newton’s method to Eq. (18), we obtain thefollowing sequence of linear systems to solve

Fig. 1. Schematic diagram of the KARMA framework.

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3216

−5.5 −5 −4.5 −4 −3.5 −3 −2.5−12

−11

−10

−9

−8

−7

−6

−5

−4

−3a

b

c

LOG(Δ Ω)

LO

G(E

rro

r)

LINEARQUADRATICSLOPE 1SLOPE 2

Spatial accuracy in ρ

−5.5 −5 −4.5 −4 −3.5 −3 −2.5−3

−2

−1

0

1

2

3

4

5

6

LOG(Δ Ω)

LO

G(E

rro

r)

LINEARQUADRATICSLOPE 1SLOPE 2

Spatial accuracy in ρΕ

−5.5 −5 −4.5 −4 −3.5 −3 −2.5−2

−1

0

1

2

3

4

5

6

LOG(Δ Ω)

LO

G(E

rro

r)

LINEARQUADRATICSLOPE 1SLOPE 2

Spatial accuracy in Fuel Temperature

Fig. 2. Conjugate heat transfer example: Spatial convergence.

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 17

J�U[�dU ¼ �F

�U[�; (19a)

U[þ1 ¼ U[ þ dU; (19b)

where [ is the Newton iteration index and JðU[Þ ¼ vFðUÞvU

���U[

is the

Jacobian matrix evaluated at U[. The linear system in Eq. (19) issolved by means of by means of a Krylov technique in which theJacobian matrix is never explicitly formed (hence the terminologyof Jacobian-free). To facilitate the Krylov linear solve, a right-preconditioned algorithm is used:�J�U[�P�1

�w ¼ �F

�U[�; (20a)

U[þ1 ¼ U[ þ P�1w; (20b)

with P the preconditioning matrix. The linear solves are per-formed using a Krylov subspace method (here, we employ GMRessince J is usually not symmetric for multiphysics problems) and theaction of JP�1 on a Krylov vector is obtained using the followingfinite-differenced definition for the matrix-vector product:

J�U[�P�1vx

F�U[ þ εP�1v

�� F�U[�

ε

; for any vector v; (21)

where 3 is a parameter used to control the magnitude of theperturbation; for additional details on JFNK technique, we referthe reader to (Knoll and Keyes, 2004; Brown and Saad, 1990). It isimportant to note that this procedure requires only the evalua-tion of the nonlinear residuals F and the entries of the Jacobianmatrix are not needed. The computational cost of the JFNKtechnique mostly resides in evaluating the nonlinear vector-valued functions that are needed at every Krylov iteration. Toreduce the dimension of the Krylov subspace, a right-preconditioner P has been introduced; this lessens both thememory and computational costs of the GMRes iterations. Thematrix-free nature of this approach relies on (i) the fact thatKrylov solvers build a solution subspace using matrix-vectoroperations only, (ii) these matrix-vector operations can beapproximated using a finite difference formula that does notrequire knowledge of the matrix entries, and (iii) the Krylovmethod is insensitive to the matrix-vector product approxima-tion. The coupling in between the various physics components isembedded in the nonlinear function f that is part of the defini-tion of F, see Eq. (18).

The preconditioner matrix P is generally chosen to be a goodapproximation of the Jacobianmatrix and should be easier to formand to solve as compared to the Jacobian matrix itself. Typically,a block-preconditioner is employed, where a single block repre-sents a given physics, and the off-block-diagonal coupling termsare ignored (akin to a block-Jacobi preconditioner); block-Gauss-Seidel preconditioning can also be employed, thereby retainingsome additional coupling in the preconditioner itself. These pre-conditioners are based on the individual physics and fall into thegeneral category of physics-based preconditioners often appliedin the JFNK approach; see also (Knoll et al., 2003; Mousseau et al.,2000).

At each Newton step, the solution of the linear solve, Eq. (19) orEq. (20), does not need to be obtained with high accuracy, notablyin the earlier steps of the nonlinear solve (when the Newton line-arization point U[ still lies far from the solution of the nonlinearproblem) (Knoll and Keyes, 2004; Lowrie, 2004; Tabesh and Zingg,2009). In inexact Newton methods, the convergence of the lineariteration is given by the following criterion

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3218

kJ�U[�P�1w þ F

�U[�k2 < gkF�U[

�k2; (22)

where the forcing term g determines how accurately the linearsolves are carried out. Variable values of g can be employed as thenonlinear solution is approached, resulting in a looser lineartolerance when one is still far from the nonlinear solution anda tighter linear tolerance when one is near the nonlinear solution;we employ the strategy given by Eisenstat and Walker (Eisenstatand Walker, 1996) for selecting g. In Section 6, we will considerthe effect of the nonlinear error on the convergence orders of thetemporal methods.

4. A multiphysics software perspective

Next, we describe the multiphysics platform from a softwareperspective. Some of the software requirements we have choseninclude: re-use of existing libraries to minimize development time,flexible data containers to write the nonlinear residuals fm(1 � m � M, where m is the single physics component index), theability to use, within the same architecture, the OS or the JFNKapproach separately for comparison purposes (since OS schemesare typically preconditioners in the JFNK approach, this require-ment is equivalent to using OS schemes as solvers rather than aspreconditioners).

The philosophy behind the software framework for multiphysicsapplications is to tightly solve coupled physical phenomena (i.e.,avoiding the traditional OS schemes) using a “loosely coupled”software methodology. The loosely coupled architecture is made

a

c

Fig. 3. Conjugate heat transfer exa

possible by requiring a software contract or a defined set ofmethodsto be implemented. For instance, methods for updating thenonlinear residual, computing the Jacobian matrix (optional), andthe preconditioner matrix provide the necessary objects needed tosolve a multiphysics application. This effort has led to the develop-ment of the KARMA framework (c(K)ode for Analysis of Reactor andother Multiphysics Applications). Other examples of computationalmultiphysics frameworks can be found in (Gaston et al., 2009;Stewart and Edwards, 2004; Lawlor et al., 2006; Jiao et al., 2006).

KARMA is a fully implicit coupled multiphysics transient analysistest-bed code and currently provides the following features that arerepresentative of current trends in coupled multiphysicsframeworks:

1. state-of-the-art computer science toolbox and libraries forefficient spatial discretizations, handling and interfacing of thevarious physics components, fast and robust linear algebra forparallel computing platforms;

2. coarse grain physics models for rapid prototyping, testing andverification; finer grain models can replace the existing ones ina straightforward fashion through the common Cþþinterfaces;

3. several coupling methodologies (simple OS, OS with Picarditerations, Strang OS (Strang, 1968), Marchuk OS (Marchuk,1971), JFNK).

KARMA ’s plug-in architecture makes it easy to modify, add, andcouple additional physics components. It also serves as a framework

b

d

mple: Temporal convergence.

10−6

10−4

10−2

100

102

104

10a6

L2 E

rro

r

OS−BE1(1)OS−SDIRK2(2)JFNK−SDIRK2(2)JFNK−SDIRK3(3)

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 19

to conduct experiments on code architectures and software design. Aprime concern in the code design is to achieve a high level of effi-ciency while still maintaining the object-oriented philosophy. Wehave employed well tested linear algebra and other general purposelibraries in order to reduce the overhead in design and imple-mentation. Similar motivations have also led to the recent develop-ment of other multiphysics codes (Gaston et al., 2009; Stewart andEdwards, 2004; Lawlor et al., 2006; Jiao et al., 2006). Some of thesupporting libraries used by KARMA include PETSc (Balay et al.,1997) (linear and nonlinear algebra library with parallel data struc-tures based on MPI), libmesh (Kirk et al., 2006) (finite elementlibrary),SLEPc (Hernandez et al., 2005) (suite of eigensolvers),Gmsh(Geuzaine and Remacle, 2009) (mesh generation), ParMETIS

(Kumar and Kumar, 1998) (parallel domain decomposition), and thevisualization software ViSit (Schroeder et al., 1998). A schematicdiagram showing the different parts of the KARMA framework and itsinteraction with the above mentioned packages is shown in Fig. 1.

A new physics model can be straightforwardly added into theKARMA framework simply by deriving from the KARMAPhy-

sicsBase class and implementing the three operators that areessential to solve the physics. These operators are

1. SystemOperator: Implement the nonlinear residual f, i.e., thespatially discretized PDE representing the physics. This oper-ator may also provide Jacobian matrix if desired; otherwisea Jacobian-free solve is performed;

2. PreconditionerOperator: Implement a preconditioner whereeither the matrix P is formed in memory or the action of P�1 isprovided in order to reduce the total number of linear itera-tions required at each Newton iteration;

3. MassOperator: Implement a mass matrix, or a lumped versionof it, for the time implicit non-autonomous PDE.

Once a physics system implements these three operators, theframework has all the necessary information required to solve theproblem. All the material properties (cross-section data, conduc-tivity, equation of state, etc.) are given as function pointers and thusarbitrary user-specified material properties can be chosen.

101 102 103 10410−8

b

CPU Time

JFNK−SDIRK5(4)

Total energy

101 102 103 10410−1

100

101

102

103

CPU Time

L2 E

rro

r

OS−BE1(1)OS−SDIRK2(2)JFNK−SDIRK2(2)JFNK−SDIRK3(3)JFNK−SDIRK5(4)

Temperature

Fig. 4. Conjugate heat transfer example: Efficiency plots.

5. Code verification

Typically, the accuracy and convergence of numericalmodels areestablished by using simplified problems for which analyticalsolutions are available. For time-dependent coupled multiphysicsproblems, analytical solutions are more difficult to obtain and weneed to resort to the Method of Manufactured Solutions (MMS)(Roache, 2002; Knupp and Salari, 2002) or semi-analytical methods(Ganapol, 2008). In the currentwork, theMMS is used extensively toanalyze and prove code correctness. The basic philosophy behindMMS is that an exact solutionUref with enough smoothness in spaceand time is chosen and substituted into the PDE to obtain a suitableforcing function (i.e., right-hand-side of the PDE) that is thenemployed in the numerical simulation. Therefore, the numericallyobtained values can be compared to the exact ones, providinga measure of the error. This procedure can be applied to nonlinearand coupled physics problems and is an useful tool for code verifi-cationpurposes. Examples areprovided inSection6. Theglobal errorin a numerical solution Unum is usually measured in the L2 norm as

kErrork2¼kUref�Unumk2¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1jUjZU

�Uref ðrÞ�UnumðrÞ

�2dr2

vuut ; (23)

where U is the spatial domain. A method is of order ps in space andorder pt in time if the error varies as OðDrps Þ and OðDtpt Þ,

respectively. The order of convergence in space is measured bycomputing the global error in a transient simulation for which thespatial mesh is successively refined. A small temporal grid isnecessary to ensure that the temporal error is small enough so thatthe error observed is due to the spatial grid only. A similar proce-dure holds for the temporal error calculation, where the spatialmesh is fine enough so that the spatial errors do not pollute theglobal error and successive simulations are performed with varioustime step sizes. The measurement of the error and the comparisonof the space/time accuracy orders obtained with expected conver-gence rates prove that the code implementation is consistent withthe mathematics and that the numerical solution converges to thetrue solution of the PDEs.

6. Results

Manufactured solutions are devised for two coupled physicsexamples: (1) a conjugate heat-transfer problem and (2) a coupledneutronics and heat conduction problem. The MMS forcing func-tions used in the simulations are given in Appendices B and C forthe different coupled physics scenarios. The accuracy achieved

101 102 103 10410−2

10−1

100

101

102

103

104

105

CPU Time

L2 E

rro

r

1e−21e−8

101 102 103 104100

101

102

103

CPU Time

L2 E

rro

r

1e−21e−8

SDIRK2(2)

101 102 103 10410−5

100

105

CPU Time

L2 E

rro

r

1e−21e−8

101 102 103 104100

101

102

103

CPU Time

L2 E

rro

r

1e−21e−8

SDIRK3(3)

101 102 103 10410−8

10−6

10−4

10−2

100

102

104

CPU Time

L2 E

rro

r

1e−21e−8

101 102 103 10410−3

10−2

10−1

100

101

102

103

CPU Time

L2 E

rro

r

1e−21e−8

SDIRK5(4)

a

b

c

Fig. 5. Influence of the nonlinear tolerance on the accuracy attained versus CPU time for the conjugate heat transfer example. Left column: total fluid energy; right column: fueltemperature.

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3220

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 21

using the tightly coupled JFNK solver and the traditional OSstrategy are compared and efficiency results (accuracy versus CPUtime) are shown for problems with disparate time scales.

6.1. Coupled conjugate heat transfer example

6.1.1. Manufactured solutionConjugate heat transfer between a conducting solid and a fluid

is modeled. The flow is assumed to be turbulent with a constantinlet mass flux; Blasius’ correlation is used to evaluate the frictionfactor fw ¼ 0.3164/Re0.25, where Re is the Reynolds number of thefluid. Energy is supplied to the fluid at the wall surface (fluidesolidinterface). The exact solutions for the fuel and fluid temperatures,Tfuel and T, on a rectangular domain are given below.

Tfuelðx; z; tÞ ¼ rF�1þ tanh

�Ctf t

���12þ sin

�px2Lx

��1þ tanh

�2w3

�wzLz

þ Tf0 ð24Þ

Tðz;tÞ ¼ rT�1þ tanh

�Ctct

���aþbtanh

��cwþwz

L

(25)

−4.5 −4 −3.5 −3 −2.5 −2 −1.5−15

−10

−5

0a

b

LOG(Δ Ω)

LO

G(E

rro

r)

LINEARQUADRATICSLOPE 1SLOPE 2

Spatial accuracy - Fast Flux

−4.5 −4 −3.5 −3 −2.5 −2 −1.5−14

−12

−10

−8

−6

−4

−2

0

LOG(Δ Ω)

LO

G(E

rro

r)

LINEARQUADRATICSLOPE 1SLOPE 2

Spatial accuracy - Fuel Temperature

Fig. 6. Spatial accuracy, coupled neutronics/heat conduction.

Z

rðz; tÞ ¼ rc þ f�1� Tðz; tÞ

Tc0

þ g

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1� Tðz; tÞ

Tc0

s: (26)

The internal energy and total energy are given by

re ¼ rCvT (27)

rE ¼ reþ 12G2

r; (28)

where rT, rF, Tf0, Tc0Ctf, Ctc,w, a, b, c, f, g, are parameters to control thespatial variations and time scales of the solution. Cv is the specificheat at constant volume and G is themass flow rate. The equation ofstate employed to close the system of equations is of the form

P ¼ P0 þ aðr� r0Þ þ bðT � T0Þ; (29)

where a, b are constant parameters. Note that a is related to thespeed of sound in the flowing medium and provides a simple wayto alter the Mach number in the calculations.

Based on this manufactured solution, the forcing functions havebeen generated and a study of the convergence order has beencarried out for various levels of spatial and temporal discretizations.The numerical solutions approach the true solution as the spatialand temporal meshes are refined, as expected. The convergencegraphs, shown in Figs. 2 and 3, prove that the implementation ofthe physics is verified and demonstrate that high-order accuracy isobtained using the JFNK technique.

6.1.2. EfficiencyThe conjugate heat transfer problem introduced above is used to

test the efficiency of the coupling methods. Here, we vary thetemporal scales of the exact solutions by changing the timeconstants such that Ctf ¼ 1000Ctc. This forces the temporal evolu-tion of the fuel temperature to occur at a faster rate than that of thefluid. In order to resolve this stiffness, we use the traditional OScoupling strategy with BE and SDIRK2(2) temporal integrationmethods and JFNK tight coupling strategy with SDIRK2(2),SDIRK3(3), and SDIRK5(4) temporal methods. The accuracy resultsobtained for a fixed spatial mesh are plotted against the totalcomputational time (work) in Fig. 4 (each curve was obtained byrunning various time step sizes to reach a given final time).

Fig. 4 presents the efficiency plots. The results indicate that thetight (JFNK) coupling scheme is optimal to obtain accurate solutionfields in all physics, once the variation in fuel temperature variableis captured In Fig. 4 we observe that the JFNK strategy (with high-order time integration) asymptotically reduces the error in thesolution more quickly (for high accuracy, i.e., small time step sizes,the JFNK-SDIRK3(3) and JFNK-SDIRK5(4) solutions are orders ofmagnitude better). In contrast, for low requirements on accuracy(larger time step sizes), the JFNK approach require more compu-tational time than OS schemes, especially OS-SDIRK2(2). However,the rate of convergence of OS-SDIRK2(2) is quite poor, equivalent tothat of a first-order scheme (OS-BE). In the plot for the temperaturevariable, Fig. 4, we note that for low CPU time values (i.e., whenlarge time step sizes are used), the temperature variable is notresolved and no schemes are accurate. Once the dynamical physicalscales have been resolved, JFNK strategies with high-order timeintegration become more efficient as the time step sizes becomesmaller.

In all of the JFNK results shown in Fig. 4, a tight nonlineartolerance was employed for the inexact Newton solverðkFðU[Þk2 < 10�10Þ. Next, to investigate the effect of this tolerance

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3222

on the efficacy using the JFNK coupling strategy, simulations areperformed using relaxed values: 10�2, and 10�8. The results areprovided in Fig. 5(a) for SDIRK2(2), Fig. 5 for SDIRK3(3) and Fig. 5(b)for SDIRK5(4). The plots indicate that increasing the nonlineartolerance in general, increases the global accuracy in the solutioneven though the convergence trends remain similar. For coarsenonlinear tolerances, the convergence in the energy variable is notguaranteed to be monotonic. As the tolerances are made morestringent, the theoretical rates of convergence for all the methodsare restored. For the temperature variable, increasing the accuracy

−8.5 −8 −7.5 −7 −6.5 −6 −5.5 −5 −4.5−18

−16

−14

−12

−10

−8

−6

−4a

b

c

LOG(Δ t)

LO

G(E

rro

r)

BE1(1)SDIRK2(2)SDIRK3(3)SLOPE 1SLOPE 2SLOPE 3

Temporal accuracy - Fast Flux

−8.5 −8 −7.5 −7 −6.5 −6 −5.5 −5 −4.5−20

−15

−10

−5

LOG(Δ t)

LO

G(E

rro

r)

BE1(1)SDIRK2(2)SDIRK3(3)SLOPE 1SLOPE 2SLOPE 3

Temporal accuracy - Thermal Flux

−8.5 −8 −7.5 −7 −6.5 −6 −5.5 −5 −4.5−30

−25

−20

−15

−10

−5

0

LOG(Δ t)

LO

G(E

rro

r)

BE1(1)SDIRK2(2)SDIRK3(3)SLOPE 1SLOPE 2SLOPE 3

Temporal accuracy - Fuel Temperature

Fig. 7. Temporal accuracy, coupled neutronics/heat conduction.

only increases the total computational cost in general in theresolved regime and does not yield any gain.

6.2. Coupled neutronics/heat conduction example: method ofmanufactured solutions

Making use of the MMS technique once again, a test problemwas constructed to verify convergence. Two neutron energy groups,two delayed neutron precursor groups and one scalar temperaturefield are considered for this coupled neutronics/heat conductionproblem. The total number of solution fields is five. The exactsolutions used in 2-D are

f1ðx; y; tÞ ¼ Af

�1þ tanh

�rft��

sin ðpxÞsin ðpyÞxy (30)

Fig. 8. Non-conforming meshes, neutronicseheat conduction problem.

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 23

f2ðx; y; tÞ ¼ f1ðx; y; tÞ�Ss;1/2�

2� (31)

101 102 103 104 10510−8

10−6

10−4

10−2

100

102

104

106

CPU Time

L2 E

rro

r

OS−BE1(1)OS−SDIRK2(2)JFNK−SDIRK2(2)JFNK−SDIRK3(3)JFNK−SDIRK5(4)

Fast Flux

a

Srem;2 þ D2Bg

Ciðx; y; tÞ ¼ Ciðx; y;0Þe�lit þZ t

0ds eliðs�tÞ Xg¼2

g¼1

bi;gnSf ;gfgðx; y; sÞ

(32)

Tðx; y; tÞ ¼ AT ð1þ tanh ðrT tÞÞsin ðpxÞsin ðpyÞ; (33)

where B2g ¼ ðp=LXÞ2 þ ðp=LY Þ2 is the geometric buckling term, Lxand LY are the domain sizes in the x and y dimensions. Af; rf AT andrT are constant parameters. Using the exact solutions for the fluxes,the exact solutions for the precursors, Ci, can easily be obtained. Theinitial precursors values are given by

Ciðx; y;0Þ ¼ 1li

Xg¼2

g¼1

bi;gnSf ;gfgðx; y;0Þ; i ¼ 1;2: (34)

Doppler feedback is accounted for in the neutronics modelthrough the removal cross section of group 1:

101 102 103 104 10510−8

10−6

10−4

10−2

100

102

104

106

CPU Time

L2 E

rro

r

OS−BE1(1)

OS−SDIRK2(2)

JFNK−SDIRK2(2)

JFNK−SDIRK3(3)

JFNK−SDIRK5(4)

Thermal Flux

101 102 103 104 10510−14

10−12

10−10

10−8

10−6

10−4

10−2

100

102

CPU Time

L2 E

rro

r

OS−BE1(1)OS−SDIRK2(2)JFNK−SDIRK2(2)JFNK−SDIRK3(3)JFNK−SDIRK5(4)

Fuel Temperature

b

c

Fig. 10. Coupled neutronics/heat conduction problem: Efficiency plots (L2 error vscomputational work).

−4.5 −4 −3.5 −3 −2.5 −2 −1.5 −1 −0.5−15

−10

−5

0a

b

LOG(Δ Ω)

LO

G(E

rro

r)

LINEARQUADRATICSLOPE 1SLOPE 2

Spatial accuracy - Fast Flux

−4.5 −4 −3.5 −3 −2.5 −2 −1.5 −1 −0.5−16

−14

−12

−10

−8

−6

−4

−2

0

LOG(Δ Ω)

LO

G(E

rro

r)

LINEARQUADRATICSLOPE 1SLOPE 2

Spatial accuracy - Fuel Temperature

Fig. 9. Spatial accuracy, coupled neutronics/heat conduction with non-conformingmeshes.

101 102 103 104 10510−4

10−2

100

102

104

106

CPU Time

L2 E

rro

r

a

b

c

1e−21e−8

101 102 103 104 10510−6

10−4

10−2

100

102

104

CPU Time

L2 E

rro

r1e−21e−8

101 102 103 104 10510−12

10−10

10−8

10−6

10−4

10−2

100

CPU Time

L2 E

rro

r

1e−21e−8

Fig. 12. Influence of the nonlinear tolerance on the accuracy attained versus CPU time(L2 error vs work) using JFNK-SDIRK3(3).

a

b

c

Fig. 11. Influence of the nonlinear tolerance on the accuracy attained versus CPU time(L2 error vs work) using JFNK-SDIRK2(2).

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3224

101 102 103 104 10510−8

10−6

10−4

10−2

100

102

104

CPU Time

L2 E

rro

r

1e−21e−8

101 102 103 104 10510−8

10−6

10−4

10−2

100

102

CPU Time

L2 E

rro

r

1e−21e−8

101 102 103 104 10510−14

10−12

10−10

10−8

10−6

10−4

10−2

CPU Time

L2 E

rro

r

1e−21e−8

a

b

c

Fig. 13. Influence of the nonlinear tolerance on the accuracy attained versus CPU time(L2 error vs work) using JFNK-SDIRK5(4).

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 25

Srem;1ðTÞ ¼ Srem;1ðT0Þ þvSrem

vffiffiffiT

p���0

� ffiffiffiT

p�

ffiffiffiffiffiT0

p �: (35)

The following equation is employed to describe thetemperature-dependent fuel conductivity:

kðTÞ ¼ k0 þvkvT

j0ðT � T0Þ: (36)

TheMatlab script used to obtained the forcing functions is givenin Appendix C. Additional cross sections could be madetemperature-dependent using the provided script.

First, the same spatial mesh is utilized for the discretization ofboth physics components and, in the next sub-section, different(non-embedded) spatial meshes are used. Spatial accuracy is veri-fied for both cases.

6.2.1. Identical spatial meshesWith the knowledge of the exact solution profiles and the

corresponding forcing functions, a space/time convergence studyhas been carried out. The convergence plots for this example areshown in Figs. 6 and 7. It is clear from the results that the solutionfields for all physics components are high-order accurate in spaceand time. By varying the coupling coefficient vSrem=v

ffiffiffiT

pin Eq.

(35) and other free parameters such as rf, rT, 1/v1, 1/v2, stiffertransients were devised and the same convergence rates havebeen observed.

6.2.2. Non-conforming spatial meshesMultiphysics applications may require that different physics

components employ different spatial meshes. Even thougha complete study of this question is outside the scope of this paper,we provide here an example where the neutronics and the heatconduction physics utilize different spatial meshes; see Fig. 8. Notethat the meshes are not embedded (when meshes are embedded,projection and interpolation operators are relatively simple todefine). As detailed in Section 3.1.2, the spatial coupling error thatmay occur due to inadequate projection/interpolation in betweensource/target meshes can be avoided by employing a high enoughnumerical quadrature in the finite element setting. It is importantto verify then that mitigation of the interpolation errors is possibleby using high-order quadratures. In this example, the same testproblem with MMS from Section 6.2.1 is considered but differentspatial meshes are employed.

The exact solution profiles for the two physics componentsshow that their spatial scales vary differently. Without the use ofhigh-order quadratures, the interpolation error started to dominateand corrupted the measured orders of convergence. In order toeliminate this, high-order quadrature rules have been employed,(see Section 3.1.2). Fig. 9 shows that spatial convergence is recov-ered, as expected, for the MMS by using high-order quadraturerules.

6.2.3. EfficiencyThe manufactured solutions for coupled neutronics/conduc-

tion problem are used to test the efficiency of the couplingmethods with disparate time scales. By changing the timeconstants as rf ¼ 1000rT in Eqs. (30)e(33), the evolution of theneutron flux is forced to occur at a considerably faster rate. Weutilize the traditional OS coupling strategy and the tight couplingstrategy with JFNK for the L� stable time integrators SDIRK2(2),SDIRK3(3), and SDIRK5(4). The accuracy results obtained areplotted against the total computational time for the differentschemes. The plots for the efficiency of temporal resolution for the

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3226

different solution fields are shown in Fig. 10. In this analysis, thereference solutionwas computed with the fourth-order SDIRK5(4)scheme.

It is evident from the results that the higher order SDIRK3(3)and SDIRK5(4) schemes prove to be efficient as compared to OSschemes. Clearly the OS-BE coupling strategy is the least efficient.The OS-SDIRK2(2) strategy exhibits a first-order convergence,similar to that of OS-BE, even though the error constant is smaller.For coarse tolerances, OS-SDIRK2(2) proves to be relatively efficientin comparison to the tightly coupled methods with SDIRK2(2) andSDIRK3(3). However, OS-SDIRK2(2) is only first-order accurateasymptotically and is less efficient than tightly coupled schemes forfiner tolerances. All JFNK schemes present error reductions inaccordance to the expected convergence rates. These resultsemphasize the gains that can be obtained from the use of tightlycoupled strategies with high-order time integrators for multi-physics problems.

Similar to the experiments carried out for the conjugate-transferproblem, the nonlinear tolerance of the inexact Newton solver isvaried from 10�2 to 10�8. The results are plotted in Fig. 11 forSDIRK2(2), Fig. 12 for SDIRK3(3) and Fig. 13 for SDIRK5(4).

The figures indicate that, for lower-order methods (SDIRK2(2)),increasing the nonlinear tolerance does not provide significantgain in accuracy. Hence it would suffice to use coarser nonlineartolerance for such methods. But for higher-order methods,a degradation in the order of convergence is clearly noticeable forcoarser tolerances and this behavior is more pronounced as theorder of the method increases. In general, care is needed whenemploying higher-order methods with relaxed tolerances sincethey become sensitive to solving the nonlinear problem morerigorously at every time step in order to maintain the theoreticalconvergence rates.

7. Conclusions

Various coupling strategies, loose OS coupling and tight JFNKbased coupling, have been implemented in a multiphysicsframework. The verification of several coupled multiphysicsproblems was performed using manufactured solutions andhigh-order spatial and temporal convergence rates wereobserved, in accordance with the theoretically expected ordersof accuracy. Spatial discretization was based on discontinuousor continuous Galerkin Finite Elements, depending upon thenature of the PDE, and temporal integration methods usedimplicit Runge-Kutta schemes. Using tight coupling methodsremoves the dominant low-order temporal coupling error dueto the operator-split linearization and high-order time integra-tion schemes can then be implemented elegantly. An approachto tackle the spatial coupling error due to the use differentmeshes between physics components was tested and high-orderconvergence rates were recovered for multiphysics problemswith non-embedded spatial meshes. This technique relied moreheavily on accurately integrating the weak forms using high-order quadrature rules than than was required in the case ofidentical spatial meshes. These preliminary results are prom-ising and further investigation is warranted. The effect of thenonlinear solve tolerance was observed: for low (second) orderscheme, a coarse nonlinear tolerance may be sufficient but forhigh-order schemes, a tighter tolerance was required to recoverthe asymptotic convergence rate. Also for relaxed accuracyrequirements, typically the OS-SDIRK2(2) method outperformedthe tightly coupled methods in both the problems tested but

asymptotically, the method still yields first-order convergencein contrast to the usage of higher order methods based on JFNKyielding higher order rates.

Studies employing high-fidelity physical models were beyondthe scope of this study and have been left for future analyses sincethe versatility of the KARMA framework allows for different physics tobe interchanged in a straightforward manner.

Appendix A. Spatial Discretization using Galerkin FEM

A.1. Elliptic equation systems

Consider an elliptic (e.g., diffusion-reaction) system of PDE givenby

�V/

,Dðr;uÞV/

uþ cðr;uÞu ¼ q c r/

˛U; (37)

supplemented with boundary conditions and where D, c, q arethe diffusion coefficient, reaction coefficient, and the source terms,respectively. For this elliptic system of equations, we employa spatial discretization based on cG-FEM. The typical weakformulation is given as

fðUÞ ¼XK

ZKfvqg �

ZKfDðr;uÞV

/v$V

/uþ cðr;uÞvug

þ boundary terms (38)

where U is the vector of unknowns that yields the solution field uexpressed with Lagrange basis functions v. The discretization yieldsa discrete vector of nonlinear residuals, f, which are then be utilizedin the nonlinear solution procedure.

A.2 Hyperbolic equation systems

Consider a nonlinear hyperbolic conservation equation of theform

V/

, F/

ðuÞ ¼ S: (39)

For hyperbolic systems of equations, we employ a spatial dis-cretization based on dG-FEM. Because of the discontinuous natureof the finite element approximation (dG), the flux F

/ðuÞ$n/ is

undefined at the cell’s boundaries and this quantity is replaced bya numerical flux. In the current work, the Rusanov or Local Lax-Friedrichs (LLF) flux function is used (Lax, 1954). The weak formof the dG method can then be written as

fðUÞ ¼XK

ZKfvSg þ

XK

8<:ZKf F/

ðuÞ$V/

vg

�Zvk

nH�uþ;u�; n

/ �vþo9=;; (40)

where u� represents the traces of u on both sides of the elementinterface; the LLF flux Hðuþ;u�; n/ Þ is given by

H�uþ;u�; n

/� ¼ 12

nF/�

uþ�,n/ þ F

/�u��,n/ þl

�uþ�u�

�o;

(41)

with l the largest eigenvalue (in absolute value) of the Jacobianmatrix of F

/.

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 27

Appendix B. Matlab script for coupled conduction/fluid MMSproblem

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3228

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 29

Appendix C. Matlab script for coupled neutronics/conductionMMS problem

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3230

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e32 31

References

Balay, S., Gropp, W.D., McInnes, L.C., Smith, B.F., 1997. Efficient Management ofParallelism in Object Oriented Numerical Software Libraries. In: Arge, E.,Bruaset, A.M., Langtangen, H.P. (Eds.), Modern Software Tools in ScientificComputing. Birkhauser Press, pp. 163e202.

Brown, P.N., Saad, Y., 1990. Hybrid Krylov methods for nonlinear systems ofequations. SIAM Journal of Scientific and Statistical Computation 11 (3),450e481.

de Boer, A., van Zuijlen, A.H., Bijl, H., 2007. Review of coupling methods for non-matching meshes. Computer Methods in Applied Mechanics and Engineering196 (8), 1515e1525.

Eisenstat, S.C., Walker, H.F., 1996. Choosing the forcing terms in an inexact Newtonmethod. SIAM Journal of Scientific Computation 17 (1), 16e32.

Fix, G.J., 1972. Effects of Quadrature Errors in Finite Element Approximation ofSteady State Eigenvalue and Parabolic Problems. Academic Press, New York.

Ganapol, B., 2008. Analytical Benchmarks for Nuclear Engineering Applications:Case Studies in Neutron Transport Theory, Tech. Rep. OECD/NEA, Paris (.

Gaston, D., Newman, C., Hansen, G., Lebrun-Grandie, D., 2009. MOOSE: a parallelcomputational framework for coupled systems of nonlinear equations. NuclearEngineering and Design 239 (10), 1768e1778.

Geuzaine, C., Remacle, J.-F., 2009. Gmsh: a 3-D finite element mesh generator withbuilt-in pre- and post-processing facilities. International Journal for NumericalMethods in Engineering 79 (11), 1309e1331.

Grandy, J., 1999. Conservative remapping and region overlays by intersectingarbitrary polyhedra. Journal of Computational Physics 148 (2), 433e466.

Hairer, E., Wanner, G., 1996. Solving Ordinary Differential Equations II: Stiff andDifferential-Algebraic Problems of Computational Mathematics. Springer Series,New York. vol. 14.

Hernandez, V., Roman, J.E., Vidal, V., 2005. SLEPc: a scalable and flexible toolkit forthe solution of Eigenvalue problems. ACM Transaction on Mathematical Soft-ware 31 (3), 351e362.

Hesthaven, J.S., Warburton, T., 2008. Nodal Discontinuous Galerkin Methods, ofAlgorithms, Analysis and Applications Series: Texts in Applied Mathematics.Springer, New York. vol. 54.

Jiao, X., Heath, M.T., 2004. Common-refinement-based data transfer between non-matching meshes in multiphysics simulations. International Journal forNumerical Methods in Engineering 61 (14), 2402e2427.

Jiao, X., Zheng, G., Alexander, P.A., Campbell, M.T., Lawlor, O.S., Norris, J.,Haselbacher, A., Heath, M.T., 2006. A system integration framework for coupledmultiphysics simulations. Engineering with Computers 22 (3), 293e309.

Kirk, B.S., Peterson, J.W., Stogner, R.H., Carey, G.F., 2006. libMesh: a cþþ library forparallel adaptive mesh refinement/coarsening simulations. Engineering withComputers 22 (3e4), 237e254.

Knoll, D.A., Keyes, D.E., 2004. Jacobian-free Newton-Krylov methods: a survey ofapproaches and applications. Journal of Computational Physics 193 (2),357e397.

Knoll, D.A., Chacon, L., Margolin, L.G., Mousseau, V.A., 2003. On balanced approxi-mations for time integration of multiple time scale systems. Journal ofComputational Physics 185 (2), 583e611.

Knupp, P., Salari, K., 2002. Verification of Computer Codes in Computational Scienceand Engineering (DiscreteMathematics and Its Applications). Chapman andHall.

Kumar, G.K., Kumar, V., 1998. Vipin, A parallel algorithm for multilevel graph par-titioning and sparse matrix ordering. Journal of Parallel and DistributedComputing 48, 71e95.

Lawlor, O., Chakravorty, S., Wilmarth, T., Choudhury, N., Dooley, I., Zheng, G., Kal, L.,2006. Parfum: a parallel framework for unstructured meshes for scalabledynamic physics applications. Engineering with Computers 22 (3), 215e235.

Lax, P.D., 1954. Weak solutions of nonlinear hyperbolic equations and theirnumerical computation. Communications on Pure and Applied Mathematics 7(1), 159e193.

Lowrie, R.B., 2004. A comparison of implicit time integration methods for nonlinearrelaxation and diffusion. Journal of Computational Physics 196, 566e590.

Mahadevan, V.S., 2006. Nonlinearly consistent schemes for coupled problems inreactor analysis, Master’s thesis, Texas A&M University, College Station, TX.

V.S. Mahadevan et al. / Progress in Nuclear Energy 55 (2012) 12e3232

Marchuk, G.I., 1971. On the theory of the splitting-up method. In: NumericalSolution of Partial Differential Equations. Academic Press, New York. vol. II.

Mousseau, V.A., Knoll, D.A., Rider, W.J., 2000. Physics-based preconditioning and theNewton-Krylov method for non-equilibrium radiation diffusion. Journal ofComputational Physics 160 (2), 743e765.

Mousseau, V.A., 2004. Implicitly balanced solution of the two-phase flow equationscoupled to nonlinear heat conduction. Journal of Computational Physics 200(1), 104e132.

Norsett, S.P., 1986. Local error control in SDIRK-methods. BIT 26 (1), 100e113.Ober, C.C., Shadid, J.N., 2004. Studies on the accuracy of time-integration methods

for the radiation-diffusion equations. Journal of Computational Physics 195 (2),743e772.

Ragusa, J.C., Mahadevan, V.S., 2009. Consistent and accurate schemes for coupledneutronics thermal-hydraulics reactor analysis. Nuclear Engineering and Design239 (3), 566e579.

Roache, P.J., 2002. Code verification by the method of manufactured solutions.Journal of Fluids Engineering 124 (1), 4e10.

Roache, P.J., September, 2009. Fundamentals of Verification and Validation. Her-mosa Publishers.

Ropp, D.L., Shadid, J.N., Ober, C.C., 2004. Studies of the accuracy of time integrationmethods for reaction-diffusion equations. Journal of Computational Physics 194(2), 544e574.

Schroeder, W.J., Martin, K.M., Lorenson, B., 1998. The Visualization Toolkit, first ed.Prentice-Hall, Englewood Cliffs, NJ.

Solin, P., Segeth, K., Dolezel, I., 2003. Higher-order Finite Element methods, CzechRepublic Series: studies in Advanced Mathematics. Chapman and Hall/CRCPress.

Solin, P., Dubcova, L., Kruis, J., 2010. Adaptive hp-fem with dynamical meshes fortransient heat and moisture transfer problems. Journal of Computational andApplied Mathematics 233 (12), 3103e3112.

Stewart, J.R., Edwards, H.C., 2004. A framework approach for developing paralleladaptive multiphysics applications. Finite Elements in Analysis and Design 40(12), 1599e1617.

Strang, G., 1968. On the construction and comparison of difference schemes. SIAMJournal on Numerical Analysis 5 (3), 506e517.

Tabesh,M., Zingg, D.W., January 5-8, 2009. Efficient implicit time-marchingmethodsusing a Newton-Krylov algorithm. In: 47th Aerospace Sciences Meeting.

Timothy, T.J., Caceres, A., 2009. Scalable parallel solution coupling for multiphysicsreactor simulation. Journal of Physics: Conference Series. doi:10.1088/1742-6596/180/1/012017. Vol.180, 012017.

Tautges, T.J., 2004. MOAB-SD: integrated structured and unstructured meshrepresentation. Engineering with Computers 20 (3), 286e293.

Zienkiewicz, O.C., Taylor, R.L., Zhu, J., 2005. The Finite Element Method, sixth ed.Butterworth-Heinemann.