High Performance Computing for Energy EfficientBuildings

Imran Akhtar∗

Interdisciplinary Center forApplied Mathematics, MC0519

Virginia Tech, Blacksburg,VA24061, USA

akhtar@vt.edu

Jeff Borggaard†

Interdisciplinary Center forApplied Mathematics, MC0519

Virginia Tech, Blacksburg,VA24061, USA

jborggaard@vt.edu

John A. Burns‡

Interdisciplinary Center forApplied Mathematics, MC0519

Virginia Tech, Blacksburg,VA24061, USA

jaburns@vt.edu

ABSTRACTCommercial buildings are the largest single consumer of en-ergy in the United States. Energy efficient buildings willhave a significant impact on overall energy consumptionand greenhouse gas emissions. Buildings being multi-scale,multi-physics, highly uncertain dynamic systems, its energyefficiency is directly linked with the design and control ofvarious systems in buildings. Achieving substantial levelsof energy savings over the life-time of a building requirenot only the state-of-the-art hardware technology but alsoa thorough computational framework which includes math-ematical algorithms, computational science methodologiesand computer tools targeted for rapid analysis, optimiza-tion and control. Direct application of high fidelity simula-tion models to problems of optimal design and control is notfeasible. Thus, reduced-order models are often developed foran efficient design and control. In this study, we present theapplication of high performance computing tools to performhigh fidelity flow simulations in a typical room which servesas a basic unit in a building. Using a large data set of theflow and temperature field distributed among various proces-sors, we compute optimal basis functions in parallel. Thesebasis functions are used in developing reduced-order modelsof complex systems for control and optimization purposes.

KeywordsHigh performance computing, reduced-order modeling, properorthogonal decomposition

1. INTRODUCTIONWith ever increasing power and speed of computational tech-nology, high performance computing (HPC) is becoming

∗Research Faculty†Professor‡Hatcher Professor in the Department of Mathematics

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page. To copyotherwise, or republish, to post on servers or to redistribute to lists, re-quires prior specific permission and/or a fee.FIT ’10, 21-DEC-2010, Islamabad, PakistanCopyright c©2010 ACM 978-1-4503-0342-2/10/12. . . $10.00

more important in science and engineering applications. Inthis paper we apply HPC to the problem of simulating anddesign of high performance energy efficient buildings. Com-mercial buildings contribute a major portion of the energyconsumption and greenhouse gas emissions in the U.S. andworldwide. Buildings systems are a network of heteroge-neous components that interact nonlinearly with wide rangeof perturbations introduced at all levels. Thus, whole build-ing simulation is a significant computational challenge. Effi-cient implementation of design, control and optimization re-quire multiple simulations which often overwhelms the avail-able resources. These complicated and complex systems areoften simplified by first developing a reduced-order modeland then applying optimal design and control strategies forefficient performance.

Building systems in the commercial sector are not typicallydesigned to conserve energy. Moreover, there is no singlecomputational tool that integrates multi-level systems andproduces an optimum solution. In order to ensure guaran-teed levels of energy savings over the life-time of a building,the proposed computational framework will include math-ematical algorithms, computational science methodologiesand computer tools targeted for rapid analysis, optimiza-tion and control. The main complexity in solving and an-alyzing these physics-based models arise due to infinite de-grees of freedom or states in the system, since most of themare governed by partial-differential equations. The modelingmust accommodate all levels of fidelity ranging from physicsbased continuum models to reduced order models. Existingmethods and computer tools are typically designed to runon single processor platforms and do not take advantage ofthe natural parallelism that occurs in building systems. Assuch, existing methods must make gross model simplifica-tions that prohibit the high performance designs that canbe achieved with high fidelity models. The new algorithmsfor high fidelity modeling will use high performance parallelcomputing for simulation, parallel optimization and sensitiv-ity analysis. These tools will enable the rapid and accurateevaluation of energy flows and provide a methodology to im-prove efficiency through integrated optimization-based tools[6, 7].

The ultimate goal is the development of high performancecomputing (HPC) tools for high performance energy effi-cient buildings to meet the challenges in energy sector. Inthe United States, the potential payoff of this effort is huge.

For example, a 50% reduction in building energy usage isequivalent to taking every passenger vehicle and small truckin the United States off the road. A 70% reduction in build-ings’ energy usage is equivalent to eliminating the entire en-ergy consumption of the U.S.A. transportation sector. Suchlevels of energy reduction have been demonstrated for one-of-a-kind designs. However, much research is needed to de-velop a new framework allowing for the development of asuite of HPC tools [8, 12] suitable for application to a broadbuilding market as illustrated in Fig. 1.

! "#$%&'(# #)%&'(# "#)%&'(#

*(&+,-./#

$0.12,+#

34561&42.4-#64#

,4#*4/656/7,+#

8&4.9:&&2#

;7+<!8&4.#

=76+/64>#

?627+,<&4#

@0&+.#=76+/64>#

?627+,<&4(#

Figure 1: Simulation requirements in a buildings.

In this paper, we present the application of HPC tools toperform high fidelity simulations of the flow field in a typicaloffice room. We use the flow field data to compute optimalbasis functions, known as proper orthogonal decomposition(POD), in parallel. The POD modes are often used as thebasis functions for the projection of the governing equationsand development of reduced-order models for nonlinear sys-tems. Conventional POD-based model reduction is based onsingular value decomposition (SVD) or eigenvalue problem(method of snapshots [14]) computed on a single processorwhich captures the essential spatial variation existing in flowfield if sufficient modes are retained [9]. We use domain de-composition to partition the flow field data and computebasis functions locally on each processor. Subsequently, theprojection procedure is also performed in parallel to com-pute the coefficients in the reduced-order models.

2. HIGH PERFORMANCE COMPUTINGTo simulate the flow field in a room along with the tem-perature distribution, we use the following nondimensionalgoverning equations:

Continuity Equation

∇ · v = 0, (1)

Momentum Equation

∂u

∂t+ v · ∇v = −∇P +

1

Re∆v, (2)

Energy Equation

∂T

∂t+ v · ∇T =

1

Re Pr∆T +B(x, t), (3)

where v(x, t) = (v1, v2, v3) represent the Cartesian veloc-ity components; P (x, t) is the pressure; and T (x, t) is the

temperature field. Re and Pr are the Reynolds number andPrandtl number, respectively. For ease of presentation, weassume inflow is fixed and the control term is given byB(x, t) = b(x)u(t) where b(x) is a given distribution andu(t) is a thermal control input. The ideas presented here arealso apply to the Boussinesq equations, where temperatureintroduces a bouyancy force in Eqn. (2). In this paper, wefocus more on the parallel implementation of the numericalalgorithm using MPI-libraries rather than spatial and tem-poral discretization. For detailed validation and verificationof the algorithm, the readers are referred to Ref. [1].

2.1 Problem StatementWe solve Eqns. (1), (2), and (3) on a 128×128×128 grid inthe x, y, z-directions on a 4×4×3 domain to simulate the flowfield in a room. Fig. 2 illustrates a typical office room withinlet and outlet representing an HVAC system. Solving theproblem sequentially would require immense computationaltime and storage. Thus, we use the domain decompositionapproach to partition the whole grid into multiple domainswhere each domain is solved on a processor. Each processorcommunicates with its neighboring processor to pass flowvariables to compute the solution.

Figure 2: A typical office room serving as a unit ina building.

2.2 Parallel ImplementationWe distribute a 128 × 128 × 128 grid on a 32-processorplatform using two-dimensional domain decomposition asshown in Fig. 3. We employ 8 × 4 topology in the y- andz−directions, respectively, such that the load on each pro-cessor is 128 × 16 × 32. Nonuniform size of the blocks inthe domain is due to the nonuniform distribution of the gridpoints. More grid points are clustered near the wall regionto resolve the boundary and thermal layers in the flow. Wenote that increasing the number of processors will furtherincrease the performance as long as the communication over-heads do not dominate the computation performed on eachprocessor. The parallel algorithm gives close to ideal speed-up and is scalable [1]. We maintain equal load balance inthe processor topology other than the flow/thermal bound-ary conditions imposed on the walls. In Table 1, minimumand maximum indices in y- and z-directions are listed for all32 processors.

In Figure 4(a), we show a communicator topology compris-ing 32 processors in a Cartesian structure with 8 processorsin the y-direction and 4 processors in the z-direction. We de-

X

01

23

4 Y01

23

4

Z

0

1

2

3

Y

Z

X

Figure 3: Two-dimensional domain decompositionon a 32-processor platform.

Table 1: Grid point distributionmyid y(jPmin,jPmax ) z(kPmin,kPmax )

0 (1,16) (1,32)1 (17,32) (1,32)2 (33,48) (1,32)...

......

7 (113,128) (1,32)8 (1,16) (33,64)9 (17,32) (33,64)...

......

30 (97,112) (97,128)31 (113,128) (97,128)

fine three communicators for message passing to the neigh-boring processors using MPI libraries; COMM-2D, COMM-1DY, and COMM-1DZ. The first processor group, known asCOMM-2D, consists of all the processors where each proces-sor knows its neighbors in the East, West, North, and Southdirections, as shown in Figure 4(b). This allows passingthe primitive variables to the neighboring processors, as andwhen required. Any information to be send to all processors,such as MPI BCAST, is performed through this communica-tor. This group is also responsible for MPI ALL REDUCEoperations. For example, using any iterative solver, we re-quire the global tolerance to satisfy certain tolerance crite-ria. We use MPI ALL REDUCE (...,MPI MAX,...), whereeach processor identifies the local maximum tolerance andbased on the global maximum tolerance continues the itera-tive process till it satisfies the tolerance criteria. The secondgroup COMM-1DY groups the processors in the y-directionas in the case of 1-D domain decomposition; that is East andWest. It allows message passing where the communication isrequired only in the y-direction. In the third group COMM-1DZ, the processors in the z-direction are grouped togetherfor explicit communication in the z-direction; that is Northand South. . Periodicity in the processor layout, if speci-fied in the system, is defined using MPI CART CREATE.One-dimensional communicators are useful for any statisti-cal analysis required in a preferred direction, e.g. averagingthe temperature distribution along the width or height of the

room and the result is summed using MPI ALL REDUCE(..., MPI SUM,..., COMM-1DY or COMM-1DZ, ...).

0 1 2 3 4 5 6

8 9 10 11 12 13

16 17 18

7

19 20 21

14 15

2322

y

z 24 25 26 27 28 29 3130

Pi

Pw

PN

PS

PE

y

z

Figure 4: (a) 8×4 = 32-processor topology in a Carte-sian structure and (b) neighboring processors.

3. THE PROPER ORTHOGONAL DECOM-POSITION

The proper orthogonal decomposition (POD) provides a toolto formulate an optimal basis or minimum degrees of free-dom (or modes) required to represent a dynamical system[9, 5, 11, 13]. POD has been widely used to identify thecoherent structures [4] in turbulent flows and examine theirstability [10], especially in fluid-structure interaction [3] andmodel-based control design [2].

To compute the POD modes, the velocity (v) and tem-perature (T ) field data is recorded at equally spaced timeinstants. After subtracting the mean, the velocity field isassembled in a matrix W3N×S , as shown in Equation (4);each column represents one time instant or a snapshot andS is the total number of snapshots for N grid points in thedomain

W =

v(1)1 v

(2)1 . . . v

(S)1

v(1)2 v

(2)2 . . . v

(S)2

......

. . ....

v(1)N v

(2)N . . . v

(S)N

(4)

Likewise, the temperature field is also assembled in a similarmanner with a matrix size of TN×S . For a 3-D problem, thisdata set is huge and is difficult to postprocess for computingthe POD eigenfunctions (Φ).

Mathematically, we compute Φ for which the following quan-

tity is maximum:⟨

|(v,Φ) |2⟩

‖Φ‖2, (5)

where 〈·〉 denotes either an ensemble or a time average. Ap-plying variational calculus, one can show that Eqn. (5) isequivalent to a Fredholm integral eigenvalue problem repre-sented as

∫

Ω

Rij(x,x′)Φj(x′) dx′ = λΦj(x), (6)

where i, j are the indices of velocity components andRij(x,x′)

is the two-point spatial autocorrelation tensor.

Using the method of snapshots [14], the constrained opti-mization problem is equivalent to a Fredholm integral eigen-value problem represented as

∫ T

0

C(t, t′)an(t′)dt′ = λnan(t), (7)

where an are the time-dependent orthogonal POD eigen-functions and C(t, t′) is the temporal correlation tensor de-fined as

C(t, t′) =1

T(Wtr

,W)Ω =

∫

Ω

WtrW dΩ (8)

The eigenvalues λn(n = 1, 2, ..., S) represents the energy ofthe corresponding POD mode Φn. The POD modes are thencomputed as

Φin(x) =

1

Tλn

∫ T

0

Wian(t) dt (9)

In the current study, each processor records the snapshots

over the local grid points in its memory. We present Algo-

rithm 1 for computing the correlation matrix of size S×S inparallel. Each processor undergoes do-loops locally over thegrid indices (i, j, k) in its domain. At the termination of thethese loops, local summation (Σp) performed on each proces-sor is added together to compute the global sum (Σg) usingMPI ALLREDUCE(Σp,Σg, ...,MPI-SUM, ...) operation. Inaddition, this command shares the Σg with all the proces-sors in the COMM-2D communicator. Thus, each processorforms the correlation matrix CS×S locally in its memory.

Algorithm 1

do m = 1, S

do n = 1, S

Σp = 0;Σg = 0

do k = kPmin, kPmax

do j = jPmin, jPmax

do i = 1, Nx

Σp = Σp + (vmi,j,k ∗ vn

i,j,k) dΩi,j,k

enddo ! i

enddo ! j

enddo ! k

CALL MPI-ALLREDUCE(Σp,Σg, ...,MPI-SUM, ...)

Cm,n = Σg

enddo ! n

enddo ! m

As part of the algorithm introduced in the method of snap-

shots, each processor computes an in Eqn. (7). The first MPOD modes are computed using Algorithm 2. Each proces-sor performs the multiplication operation of the local snap-shot data with an to compute the POD modes. Thus, thePOD modes are distributed over the whole domain with thesame distribution topology.

Algorithm 2

do k = kPmin, kPmax

do j = jPmin, jPmax

do i = 1, Nx

do m = 1,M

Σ = 0

do n = 1, S

Σ = Σ+Wni,j,k ∗ an,m

enddo ! n

Φmi,j,k =

Σ

Sλm

enddo ! m

enddo ! i

enddo ! j

enddo ! k

4. NUMERICAL RESULTSWe simulate the flow in the 4 × 3 room with the followinginflow and outflow locations:

Γinlet : x = 0.0; 1.5 ≤ y ≥ 2.5; 2.5 ≤ z ≥ 2.75,

Γoutlet : x = 4.0; 1.75 ≤ y ≥ 2.25; 0.25 ≤ z ≥ 0.75.

We note that the numerical problem is set up to show ap-plication of HPC tools for HVAC systems, in general. Thenumerical values for the inflow and temperature may not ac-tually represent a physical cooling/heating problem ratherit may just be ventilation in a room in an isothermal setting.Parabolic inflow boundary conditions are introduced at theinlet with average velocity v = (1.0, 0.0, 0.0) while Neumannboundary conditions are applied at the outflow. The ambi-ent temperature in the room is at T = 21C while the inlettemperature is set at Tinlet = 20C. The Reynolds number,based on the average inlet velocity and width of the inlet, is100 and Prandtl number is 0.71. We perform the numericalsimulation and compute the time evolution of the velocityand the temperature fields. We plot instantaneous flow fieldat time t = 300 units of the horizontal velocity componentand the temperature field in Figs. 5 and 6, respectively. Twoprobes are placed in the domain at point (P1) close to theinlet and (P2) in the middle of the room. As shown in Fig.7, the temperature at probe (P1) settles to the inlet temper-ature quickly while the temperature slowly decays from theambient temperature towards the inlet temperature setting.

We record 200 snapshots of the flow field data as in Eqn.(4) equally spaced in time t = [10, 310]. We then com-pute the POD modes using Algorithm 1 and Algorithm 2.Based on the eigenvalues of W, the first velocity POD mode(Φv

1 = [φv11 , φ

v21 , φ

v31 ]) captures 98.47% of the total energy

in the dynamical system while the first temperature POD

X 01234 Y

01

23

4

Z

0

1

2

3

X

Z

Y

Figure 5: Horizontal velocity component (v1).

X 01234 Y

01

23

4

Z

0

1

2

3

X

Z

Y

TEMP294293.9293.8293.7293.6293.5293.4293.3293.2

Figure 6: Temperature field distribution. Note thattemperature is in Kevin scale (293K = 20C ).

Time

Tem

p

100 200 300

293

294

Figure 7: Temporal variation of the temperaturefield close to inlet (solid) and middle of the room(dashed). Note that temperature is in Kevin scale(293K = 20C ).

mode (φT1 ) has 99.88% of the total contribution. In Fig. 8,

we plot the contours of the first velocity POD mode ranging

from [-0.2,0.2]. As expected, we observe dominant struc-tures in the middle of the room mainly responsible for thedynamics of the flow field. The first temperature POD modeexhibit the modal distribution along the streamlines of theflow field as shown in Fig. 9. The POD modes computedin this study for a high resolution 3-D flow with 200 datasnapshots is made possible through the high performancecomputing tools. Consequently, the governing equations inEqn. (1), (2), and (3) can be projected onto these PODmodes in parallel to develop a reduced-order model. Thereduced-order model thus obtained can be used to designtemperature control or optimization of the HVAC system.

ConclusionsWe introduced high performance computing tools for use inenergy efficient buildings. Buildings being complex frame-work of various systems, contribute a significant portion ofenergy consumption world wide. To control the systemsin a building for efficient energy consumption, one requireshigh fidelity simulation tools which are not feasible from im-plementation view point. Reduced-order models provide areasonable alternative for design, control and optimizationof complex systems. Using parallel computing, we intend todevelop reduced-order models and improve on the design,control, and optimization techniques for building systems.In this study, we simulated a 3-D flow in a typical room of abuilding and computed the flow and temperature field usingparallel computing. Using the recorded flow data, we intro-duced parallel algorithms to compute the POD basis func-tions required in the development of reduced-order modelsof nonlinear systems. The main point of this short paper isto illustrate how HPC based simulation can be coupled ina natural way with parallel algorithms for computing PODbasis. For whole building problems, it will be essential tomake use of this parallelism to construct low order designand control models. We plan to apply these HPC methodsto a control design problem in a future paper.

AcknowledgmentsThis research was supported in part by Air Force Office ofScientific Research grants FA9550-07-1-0273 and FA9550-10-1-0201 and by the Environmental Security TechnologyCertification Program (ESTCP) under a subcontract fromthe United Technologies.

5. REFERENCES[1] I. Akhtar. Parallel Simulations, Reduced-Order

Modeling, and Feedback Control of Vortex Shedding

using Fluidic Actuators. PhD thesis, Virginia Tech,Blacksburg, VA, 2008.

[2] I. Akhtar and A. H. Nayfeh. Model based control ofvortex shedding using fluidic actuators. Journal ofComputational and Nonlinear Dynamics, 5(4):041015,2010.

[3] I. Akhtar, A. H. Nayfeh, and C. J. Ribbens. On thestability and extension of reduced-order Galerkinmodels in incompressible flows: A numerical study ofvortex shedding. Theoretical and Computational Fluid

Dynamics, 23(3):213–237, 2009.

[4] H. P. Bakewell and J. L. Lumley. Viscous sublayer andadjacent wall region in turbulent pipe flow. ThePhysics of Fluids, 10(9):1880–1889, 1967.

X 01234 Y

01

23

4

Z

0

1

2

3

X

Z

Y

X 01234 Y

01

23

4

Z

0

1

2

3

X

Z

Y

X 01234 Y

01

23

4

Z

0

1

2

3

X

Z

Y

Figure 8: The first velocity POD modes: (a) φv11 , (b)

φv21 , and (c) φ

v31 .

[5] G. Berkooz, P. Holmes, and J. L. Lumley. The properorthogonal decomposition in the analysis of turbulentflows. Annual Review of Fluid Mechanics, 53:321–575,1993.

X 01234 Y

01

23

4

Z

0

1

2

3

X

Z

Y

Figure 9: The first temperature POD mode (φT1 ).

[6] J. Borggaard, J. A. Burns, E. M. Cliff, andL. Zietsman. A pde approach to optimization andcontrol of high performance buildings. In Proceedings

of the Oberwolfach Workshop on Numerical

Techniques for Optimization Problems with PDE

Constraints. M. Heinkenschloss, R. H. Hoppe and V.Schulz, Eds. January 2009, 205–208, 2009.

[7] J. Borggaard, J. A. Burns, A. Surana, andL. Zietsman. Control, estimation and optimization ofenergy efficient buildings. In Proceedings of the 2009

American Control Conference. St. Louis, MO, June2009, 837–841, 2009.

[8] J. A. Clarke. Energy Simulation in Building Design.Butterworth Heinemann, 2001.

[9] A. E. Deane, I. G. Kevrekidis, G. E. Karniadakis, andS. A. Orsag. Low-dimensional models for complexgeometry flows: Application to grooved channels andcircular cylinder. Physics of Fluids A,3(10):2337–2354, 1991.

[10] P. Holmes, J. L. Lumley, and G. Berkooz. Turbulence,Coherent Structures, Dynamical Systems and

Symmetry. Cambridge University Press, Cambridge,UK, 1996.

[11] X. Ma and G. Karniadakis. A low-dimensional modelfor simulating three-dimensional cylinder flow. Journalof Fluid Mechanics, 458:181–190, 2002.

[12] A. Malkawi and G. Augenbroe. Advanced Building

Simulation. Routelegde, UK, 2004.

[13] B. R. Noack, K. Afanasiev, M. Morzynski, andF. Thiele. A hierarchy of low-dimensional models forthe transient and post-transient cylinder wake.Journal of Fluid Mechanics, 497:335–363, 2003.

[14] L. Sirovich. Turbulence and the dynamics of coherentstructures. Quarterly of Applied Mathematics,45:561–590, 1987.