1

G-CRS: GPU Accelerated CauchyReed-Solomon Coding

Chengjian Liu, Qiang Wang, Xiaowen Chu, and Yiu-Wing Leung

Abstract—Recently, erasure coding has been extensively deployed in large-scale storage systems to replace data replication. With theincrease in disk I/O throughput and network bandwidth, the performance of erasure coding becomes a major bottleneck oferasure-coded storage systems. In this paper, we propose a graphics processing unit (GPU)-based implementation of erasure codingnamed G-CRS, which employs the Cauchy Reed-Solomon (CRS) code, to overcome the aforementioned bottleneck. To maximize thecoding performance of G-CRS, we designed and implemented a set of optimization strategies, such as a compact structure to store thebitmatrix in GPU constant memory, efficient data access through shared memory, and decoding parallelism, to fully utilize the GPUresources. In addition, we derived a simple yet accurate performance model to demonstrate the maximum coding performance ofG-CRS on GPU. We evaluated the performance of G-CRS through extensive experiments on modern GPU architectures such asMaxwell and Pascal, and compared with other state-of-the-art coding libraries. The evaluation results revealed that the throughput ofG-CRS was 10 times faster than most of the other coding libraries. Moreover, G-CRS outperformed PErasure (a recently developed,well optimized CRS coding library on the GPU) by up to 3 times in the same architecture.

Index Terms—Cauchy Reed-Solomon Code, Graphics Processing Unit, Erasure Coding, Distributed Storage System

F

1 INTRODUCTION

THe past decade has witnessed the rapid growth of datain large-scale distributed storage systems. For example,

the European Centre for Medium-Range Weather Forecasts[1] reached a data amount of 100 PB and experienced anannual growth rate of 45%. Traditionally, data replicationscheme such as triplication is used by distributed storagesystems to protect users’ data due to its simplicity. Currently,with the rapid increase of data volume, the reduction ofstorage overhead has become an unavoidable task in large-scale storage systems. Consequently, erasure codes havebeen introduced in many storage systems because theycan provide a higher storage efficiency and fault tolerancethan data replication. Examples of such storage systemsinclude Microsoft cloud service Azure [2] and Facebook’swarehouse [3] and Web service storage system f4 [4]. Inaddition, distributed file systems such as Hadoop [5] andCeph [6] have begun to support erasure coding to yield ahigher reliability and lower storage overhead.

A general erasure coding system works as follows. Ini-tially, the user data to be protected is divided into k equalsized data chunks. The encoding operation gathers all kdata chunks and generates m equal sized parity chunksaccording to an encoding algorithm. In a distributed storagesystem, the set of n=k+m data and parity chunks are usuallystored at different hardware devices to prevent data lossdue to device failures. When no more than m devices failout of these n devices, the chunks in the failed devices

• Chengjian Liu, Qiang Wang, Xiaowen Chu and Yiu-Wing Leung arewith the Department of Computer Science, Hong Kong Baptist University,Kowloon, Hong Kong.

• Xiaowen Chu is also with HKBU Institute of Research and ContinuingEducation, Shenzhen, China.E-mail: {cscjliu,qiangwang,chxw,ywleung}@comp.hkbu.edu.hk

become unavailable. To recover the lost data, a decodingoperation gathers k available chunks and reproduces themissing chunks according to a decoding algorithm. Theerasure codes that can restore m missing chunks from theremaining k alive chunks have the highest error correctioncapability and are called Maximum Distance Separable (MDS)codes [7]. Reed-Solomon (RS) coding [8] and its variantCauchy Reed-Solomon (CRS) coding [9] are the two well-known general MDS codes that can support any values of kand m.

In erasure coded storage systems, an encoding operationis triggered when data is written on the storage system,whereas decoding is performed when some lost data isrequired to be recovered. Thus, the encoding and decodingperformance is crucial for the quality of service and userexperience. The coding performance is usually inverselyproportional to m. Modern data centers begin to deploy 40-100Gb/s Ethernet or even InfiniBand FDR/QDR/EDR toimprove the network speed [10], and disk arrays based onSolid-State Drives (SSD) to improve the disk input-output(I/O) performance [11]. Such technology trend pushes thecomputationally expensive erasure coding into a potentialperformance bottleneck in erasure-coded storage systems.

Recently, graphics processing units (GPUs) have beenused in some storage systems to perform different com-putationally expensive tasks. Shredder [12] is one frame-work used for leveraging GPUs to efficiently chunk filesfor data deduplication and incremental storage. GPUstore[13] is another framework for integrating GPUs into storagesystems for file-level or block-level encryption and RAID6 data recovery. Another GPU-based RAID 6 system hasbeen developed in [14], which uses GPUs to accelerate twoRAID 6 coding schemes, namely Blaum-Roth and Liberationcodes. This system achieves a coding speed of up to 3GB/s.However, RAID 6 only supports up to two disk failures

2

and is not suitable for large-scale systems. To date, Gibraltar[15], which employs table lookup operations to implementGalois field multiplications, is the most successful GPU-based Reed-Solomon coding library; notably, the systemhas much higher speed over the single-thread Jerasure [16],which is the most popular erasure coding library for CPUs.PErasure [17] is a recent CRS coding library for GPUs andits performance is even better than Gibraltar. However,PErasure does not fully utilize the GPU memory systemand results in sub-optimal performance. With the rapidimprovement of networking speed and aggregated disk I/Othroughput, there is a demand to further improve the codingperformance.

In this paper, we aim to design a new CRS coding libraryfor GPUs, namely G-CRS, that can fully utilize the GPUresources and deliver high coding performance that cansaturate the state-of-the-art network speed. To this end, wehave designed new data structures and a set of optimizationstrategies. G-CRS can achieve more than 50GB/s of rawcoding performance on a modern GPU Nvidia Titan X forthe case of m = 16 (i.e., the system can withstand up to 16device failures)1. Our major contributions are summarizedas follows:

1) We present a step-by-step optimization analysis to re-veal the method of utilizing GPUs to accelerate CRScoding.

2) We present a simple yet accurate performance modelto understand the major factors that affect the codingperformance.

3) We provide a pipelined mechanism to enable our G-CRS to achieve a peak performance by efficiently over-lapping data copy operations and coding operations.

4) We conduct extensive experiments to validate the effec-tiveness of our proposed G-CRS, compare with otherstate-of-the-art coding libraries, and analyze the domi-nating factor.

5) G-CRS is open-source and freely available to the public.The remainder of this paper is organized as follows. Sec-

tion 2 introduces the background of GPU computing, CRScoding, and some related work. Section 3 gives the detailsof our optimization and implementation of G-CRS. Section 4presents a performance model of G-CRS to understand theimpact of computational power, memory bandwidth, andcoding parameters on the coding performance. Section 5presents a pipelined mechanism that can further enhancethe performance of G-CRS. Section 6 presents the experi-mental results of G-CRS and other state-of-the-art libraries.We summarize the paper in Section 7.

2 BACKGROUND AND RELATED WORK

In this section, we first briefly introduce the key conceptsof GPU computing and CRS coding. Then, we identify thechallenges of applying erasure coding in modern distributedstorage systems. Finally, we introduce some related work.

2.1 GPU Computing and CUDAModern GPUs are typically equipped with hundreds tothousands of processing cores evenly distributed on several

1. The source code and experimental results of our G-CRS are avail-able at http://www.comp.hkbu.edu.hk/∼chxw/gcrs.html.

streaming multiprocessors (SMs). For example, the NvidiaGTX 980 with Maxwell architecture contains 16 SMs and4GB off-chip GDDR memory. Each SM has 128 StreamProcessors (SPs, or cores) and a 96-KB on-chip memorynamed shared memory, which has much higher throughputand lower latency than the off-chip GDDR memory [18].Besides the 2MB L2-cache shared by the whole GPU, eachSM also has a small amount of on-chip cache to speed upthe data access of read-only constant memory.

Currently, CUDA is the most popular programmingmodel for GPUs [19]. A typical CUDA program compriseshost functions, which are executed on the central processingunit (CPU), and kernel functions, which are executed onthe GPU. Each kernel function runs as a grid of threads,which are organized into many equal sized thread blocks.Each thread block can include a set of threads distributedin a number of thread warps, each of which has 32 threadsthat execute the same instruction at a time. Threads in athread block can share data through their shared memoryand perform barrier synchronization.

2.2 Cauchy Reed-Solomon Coding

1 0 … 0

0 1 … 0

… … …

0 0 … 1

x0,0 x0,1 … x0,kw-1

xmw-1,0 xmw-1,1 … xmw-1,kw-1

… … … …

kw

kw

mw

Generator matrix

Data

d1…

T

dk-1T

=

Data+Codeword

c0…

T

cm-1T

d0T

d1…

T

dk-1T

d0T

Fig. 1. Illustration of CRS encoding. |di|T is the w-row format of datachunk di. |ci|T is the w-row format of parity chunk ci.

As illustrated in Fig. 1, the encoding procedure of CRStakes k equal sized data chunks d0, d1, ..., dk−1 as input,and generates m parity chunks c0, c1, ..., cm−1 as output. Toperform the coding, it needs to select an integer parameterw that is no less that log2(k + m). Hence a CRS code canbe defined by a triple (k,m,w). CRS first defines an m × kCauchy distribution matrix over Galois Field GF (2w), andthen expands it into a (k +m)w × kw generator matrix overGF (2) whose elements are either 1 or 0 [9]. Notice that thetop kw rows of the generator matrix is an identity matrix.

Each data chunk di needs to be transformed into w rows,denoted by Di,0, Di,1, ..., Di,w−1. The w-row format of datachunk di is denoted by |di|T . Then all the k data chunks canbe combined into a data matrix with kw rows. By multiply-ing the generator matrix and data matrix over GF (2), wecan get k+m output chunks that include the k original datachunks (due to the kw× kw identity sub-matrix) and the mparity chunks. Notice that the multiplication in GF (2) canbe implemented by efficient bit-wise XOR operations, whichis the major property of CRS.

Fig. 2 presents a concrete example of the CRS encodingprocess where k=2, m=2 and w=2. Data chunk di consists oftwo rows: Di,0 and Di,1, i = 0, 1. In the actual encoding

3

1 1 0 11 0 1 10 1 1 11 1 1 0

1 0 0 00 1 0 00 0 1 00 0 0 1

D0,0D0,1D1,0D1,1

C0,0＝D0,0 Å D0,1 Å D1,1

C0,1＝D0,0 Å D1,0 Å D1,1

C1,0＝D0,1 Å D1,0 Å D1,1

C1,1＝D0,0 Å D0,1 Å D1,0

Fig. 2. A concrete example of CRS encoding. k = 2,m = 2, w = 2.

process, we only need to calculate the m parity chunksusing the bottom mw rows of the generator matrix. For eachparity chunk, the values from the corresponding row vectorin the generator matrix determine which data chunks willbe involved in the XOR operations. For example, the fifthrow vector from the generator matrix in Fig. 2 < 1, 1, 0, 1 >determines that C0,0 is generated from D0,0, D0,1 and D1,1

with XOR operations.When either a data chunk or a parity chunk becomes

unavailable due to device failure, a decoding operation istriggered to restore the missing chunk. To recover a paritychunk, w row vectors from the generator matrix togetherwith all data chunks serve as the input. By contrast, torecover a data chunk, an inverse matrix is generated fromthe generator matrix (which can be done offline in advance),and k alive chunks serve as the input with w row vec-tors from the inverse matrix. The encoding and decodingoperations are essentially the same in terms of data accesspattern and computation. Therefore, we use the term codingto represent both encoding and decoding.

2.3 Opportunities and Challenges in Erasure CodingData reliability and availability are critical requirements fordata storage systems. Although coding-based RAID 5/6have become the industry standards for decades, replica-tion is still the de facto data protection solution in large-scale distributed storage systems. With the increase in theamounts of data and the deployment of expensive SSDs,there is a great opportunity for erasure coding becauseit can provide much lower storage overhead and higherreliability compared with data replication. In [20], a com-prehensive comparison of erasure coding and replicationwas presented. However, erasure coding is a compute- anddata-intensive task which brings practical challenges to itsadoption in distributed storage systems.

In an erasure coded distributed storage system, therecould be three potential performance bottlenecks: the ag-gregated disk I/O throughput, the network bandwidth,and the coding performance. Modern data centers havestarted to deploy high-speed network switches with morethan 40Gb/s of bandwidth per network port [10]. Facebookand LinkedIn are already working on 100Gb/s network fortheir data centers [21] [22]. Meanwhile, the sequential I/Othroughput of a single SSD has been improved to more than4Gb/s [23], and the aggregated I/O throughput of a diskarray can easily match the network bandwidth. However,the throughput of software implemented erasure codingis inversely proportional to the number of parity chunks,and is typically less than 10 Gb/s on multi-core CPUs[17], which makes erasure coding impractical for large-scale

distributed systems. Modern GPUs have tens of TFlopscomputation power and an internal memory bandwidth ofa few hundreds of GB/s, providing an opportunity to speedup erasure coding to saturate the disk I/O and networkbandwidth. This motivates us to design and implement G-CRS to fully utilize the GPU power and achieve a highcoding throughput.

2.4 Related Work

Coding Performance: Many research and industrial studieshave focused on improving the performance of erasure cod-ing. One pioneer study optimizes the Cauchy distributionmatrix that results in better coding performance for CRScoding on the CPU by performing less XOR operations [24].Jerasure [16] is a popular library that implements variouskinds of erasure codes on the CPU, including optimizedCRS coding. Optimization with efficient scheduling of XORoperations on CPU is presented in [25]. These sequen-tial CRS algorithms are designed for CPUs and are notsuitable for GPUs due to their complicated control flows.Another thread of research aims to exploit parallel com-puting techniques to speed up erasure coding. For multi-core CPUs, CRS codes have been parallelized in [26], [27];EVENODD codes have been parallelized in [28]; and RDPcodes have been parallelized in [29]. A fast Galois fieldarithmetic library for multi-core CPUs with the supportof Intel SIMD instructions has recently been presented in[30]. Although these works have achieved great improve-ment, their coding performance is still not comparable tothe throughput of today’s high speed networks, especiallywhen a large number of parity data chunks are requiredfor higher data reliability. These parallel algorithms cannotbe directly applied to GPUs either, due to their differenthardware architectures. For many-core GPUs, the Gibraltarlibrary [15] implements the classical Reed-Solomon codingand outperforms many existing coding libraries on CPUs.PErasure [17] is a recent CRS coding library for GPUs andits performance is better than Gibraltar. However, PErasuredoes not fully utilize the GPU memory system and resultsin sub-optimal performance.Offloading Tasks to GPUs: Modern GPUs are embeddedwith hundreds to thousands of arithmetic processing unitsthat provide tremendous computing power, and attractsmany work to port computationally intensive applicationsfrom CPUs to GPUs. For example, G-Blastn [31], which is anucleotide alignment tool for the GPU, achieves more than10 times of performance improvement over its CPU versionNCBI-BLAST. In [32], regular expression matching on GPUscan be 48 times faster than that on CPUs. SSLShader [33] andAES [34] both demonstrate great performance improvementwhen data encryption algorithms are offloaded to GPUs.Network coding on GPUs, such as those in [35], [36] andNuclei [37], are the most closely related work in addition tothe aforementioned Gibraltar [15] and PErasure [17]. Thesestudies aim to improve the performance of network codingto match the throughput of high speed networks for bothencoding and decoding.

4

3 DESIGN OF G-CRS

In this section, we first present a high-level view of G-CRSand define our terminologies. Next, we provide a baselineimplementation of CRS coding on GPUs that directly mi-grates from the CPU version. Subsequently, we analyze thepotential drawbacks of this basic design and provide a set ofoptimization strategies to accelerate CRS coding on GPUs.Our G-CRS is implemented by applying all optimizationstrategies described in this section.

Fig. 3 illustrates the system architecture of G-CRS thatimplements a (k, m, w) CRC code. The bitmatrix stores thebottom mw rows of the generator matrix of the CRS code.The input data is divided into k equal sized data chunks, andthe output includes m parity chunks with the same size.We use s to represent the number of bytes of a long datatype on the target hardware platform, i.e., s = sizeof(long).We define a packet as s consecutive bytes. The XOR oftwo packets can then be efficiently carried out by a singleinstruction. We define a data block as w consecutive packets,where w is the parameter of CRC and should be no lessthan log2(k +m). The number of data blocks in a chunk isdenoted by N .

long *in

Input data of L bytes (e.g., a file)Input data of L bytes (e.g., a file)

long *in

Input data of L bytes (e.g., a file)

long *in

Input data of L bytes (e.g., a file)

Data Chunk 0

…

…

Data Chunk 1

Block 1

Packet 0

Packet 1

Packet w-1

......

Block 1

Packet 0

Packet 1

Packet w-1

...

Block 1

Packet 0

Packet 1

Packet w-1

...

Block 1

Packet 0

Packet 1

Packet w-1

...

Block 0

Packet 0

Packet 1

Packet w-1

......

Block 0

Packet 0

Packet 1

Packet w-1

...

Block 0

Packet 0

Packet 1

Packet w-1

...

Block 0

Packet 0

Packet 1

Packet w-1

...

Block N-1

Packet 0

Packet 1

Packet w-1

......

Block N-1

Packet 0

Packet 1

Packet w-1

...

Block N-1

Packet 0

Packet 1

Packet w-1

...

Block N-1

Packet 0

Packet 1

Packet w-1

...

Data Chunk k-1

long *out

Parity Chunk 0

Parity Chunk m-1

…

Parity Chunk 1

long *out

Parity Chunk 0

Parity Chunk m-1

…

Parity Chunk 1

kw

mw bitmatrix

kw

mw bitmatrix

Fig. 3. Input and Output for Coding

We summarize the high level workflow of G-CRS inAlgorithm 1. When the size of user data is greater than theavailable GPU memory, we will encode the data in differentrounds.

Algorithm 1 High Level Workflow of G-CRS1: Input: k, m, w, bitmatrix, dataSize2: Compute round from k, m, w, dataSize3: Construct bitmatrix4: Allocate GPU memory5: Copy bitmatrix to GPU6: for each i ∈ [1, round] do7: Copy k data chunks of the i-th round to GPU8: Launch the encoding kernel function to generate m

coding chunks9: Copy back m parity chunks of the i-th round to main

memory10: end for11: Free memory resources

3.1 Baseline Implementation

Listing 1 presents the baseline implementation of the CRScoding kernel written in CUDA, which is directly migratedfrom a CPU version. The kernel function defines the behav-ior of a single GPU thread. In this implementation, eachthread is responsible for encoding a single packet. Whenthe kernel function is launched, a total of mwN threadswill be created to generate the m parity chunks in parallel.Each element of bitmatrix is represented by an integer. Thiskernel function works as follows: (1) The input buffer inand the output buffer out are located in the GPU globalmemory. (2) The mw bottom row vectors from the generatormatrix are stored in the bitmatrix, which is also located in theglobal memory. (3) Each thread calculates the initial indexof its assigned packet in each data block (line 13). Then,each thread iterates its corresponding row in the bitmatrix todetermine the data required to perform the XOR operation(lines 16-24). (4) Each thread writes a packet, which is storedin the variable result, to the output buffer (line 26).1 global void crs coding kernel naive2 ( long ∗ in , long ∗out , i n t ∗bi tmatr ix ,3 i n t s ize , i n t k , i n t m, i n t w) {4 i n t blockUnits = blockDim . x / w;5 i n t blockPackets = blockUnits ∗ w;6 i n t t i d = threadIdx . x + blockIdx . x ∗

blockPackets ;7 i n t u n i t i d o f f s e t = t i d / w ∗ w;8 i n t u n i t i n i d = t i d % w, i , j ;9

10 i f ( threadIdx . x >= blockPackets ) return ;11 i f ( t i d >= s i z e ) return ;1213 i n t index = threadIdx . y∗k∗w∗w+ u n i t i n i d ∗k∗w;14 long r e s u l t = 0 , input ;1516 for ( i = 0 ; i < k ; ++ i ) {17 for ( j = 0 ; j < w ; ++ j ) {18 i f ( b i t m a t r i x [ index ] == 1) {19 input = ∗ ( in+ s i z e ∗ i + u n i t i d o f f s e t + j ) ;20 r e s u l t = r e s u l t ˆ input ;21 }22 ++index ;23 }24 }2526 ∗ ( out + s i z e ∗ threadIdx . y + t i d ) = r e s u l t ;27 }

Listing 1. A baseline implementation of CRS coding on GPU

Some severe performance penalties exist in this base-line implementation, implying that the GPU resources aresubstantially under-utilized. To design a fully optimizedversion of CRS coding on GPU, a thorough understandingof the GPU architecture is required, including its memorysubsystem and its method of handling branch divergence.We have identified three major performance penalties of thebaseline implementation:

Inefficient memory access: The memory access patternof the baseline kernel implementation in its global memorycauses a considerable performance penalty. From lines 18-20 of Listing 1, we can observe that each XOR operationrequires two memory reads, one from the bitmatrix andanother one from the input buffer. Since the baseline im-plementation stores the bitmatrix and input buffer in globalmemory, each memory access to the bitmatrix or input bufferis very likely to generate a global memory read because

5

TABLE 1Features of different types of GPU memory space

Memory Access Latency Cached ScopeGlobal Around 600 cycles Yes All threads

Constant Around 400 cycles Yes All threads

Shared Around 20 cycles N/A All threads in thesame thread block

Register Around 4 cycles N/A Single thread

the GPU L2 cache is small and shared by the whole globalmemory. Accesses to global memory have long latency andlow throughput as compared to the XOR operations, hencethe performance of the baseline implementation will belimited by the global memory throughput.

Underutilization of memory bandwidth: Since an in-teger is used to store the binary value of 1 or 0 in thebitmatrix, the memory bandwidth utilization is very lowwhen accessing the bitmatrix. Regarding the input data, eachdata packet is in fact accessed by mw threads, implying anopportunity of temporal reuse. But due to the small size ofGPU L2 cache and large data size, the L2 cache hit rate isnot satisfactory.

Penalty of Control Flow: Finally, the GPU architec-ture is inefficient at handling control flow. Specifically, aconstant performance penalty is caused by control flow.This is because the threads in a warp can only executethe same instruction simultaneously; if threads in a warpexecute different instructions due to control divergence, theexecution of these instructions are serialized. The penaltyis often incurred by if-then-else and loop statements. In thebaseline implementation, we have two for loop statementsat lines 16 and 17, and an if statement at line 18. Althoughsome threads do not require the execution of the code insidethe if statement, given a warp, they are required to wait forother threads to finish the calculation in that same warpbefore proceeding with their execution.

3.2 Optimization StrategiesIn this subsection, we present a set of optimization strategiesfor overcoming the major performance penalties discussedin the previous subsection. Here we first investigate how toimprove the performance of accessing bitmatrix. Next, weachieve efficient access to data blocks by loading them toon-chip shared memory before accessing and reducing thethread dimension. We also present our strategy of removingcontrol divergence. At last, we discuss how to make decod-ing performance the same as encoding.

3.2.1 Efficient access to the bitmatrixGPU memory system has a very complicated structure [18].Table 1 presents three major features of four types of com-monly used GPU memory. Global memory has the longestaccess latency, and registers have the shortest access latency.Although the access latency of physical constant memory isclose to that of global memory, each SM offers an on-chipcache for it, which makes it perfect to store a small amountof read-only data that are frequently accessed.

Since bitmatrix is frequently accessed by all threads, wecan store it in the constant memory to reduce access latency

because each SM offers a small on-chip cache for read-onlydata in the constant memory. The bitmatrix contains kmw2

elements. However, the size of the constant cache is limitedto the KB-level. If the size of the bitmatrix exceeds the size ofthe constant cache in each SM, the access will eventually goto the global memory and result in long access latency. Onepossible solution is to use the char data type (which has asize of 1 byte) to store each element of the bitmatrix insteadof using the int type (which has a size of 4 bytes). Thisapproach reduces the size of the bitmatrix to one-quarter ofits original size, enabling it to more easily fit in the constantcache.

Although storing the bitmatrix in constant memory re-duces the access latency, it may still suffer from the follow-ing performance penalty. If multiple threads in the samewarp access the same memory address in the constant cache,the access can proceed in parallel. However, if multiplethreads in the same warp access different memory addressesin the constant cache, these accesses will be serialized [38],incurring substantial performance penalty. We thus furtherinvestigate how w successive threads from the same warpaccess the bitmatrix in each iteration. The access pattern tothe bitmatrix in our implementation is illustrated in Fig. 4. Ineach iteration, w threads simultaneously access w differentelements. If the access latency of the on-chip memory takesone time slot, w time slots may be required to obtainelements from the bitmatrix for each iteration.

…

0

1

w-1

tid

…

x0,0

x1,0

…

xw-1,0

Iteration: j=0

0

1

w-1

tid

…

x0,1

…

Iteration: j=1

xw-1,1

x1,1

0

1

tid

…

x0,w-1

x1,w-1

…

Iteration: j=w-1

xw-1,w-1w-1

Fig. 4. Access pattern to bitmatrix. tid refers to thread ID.

To reduce the access latency to the read-only cache,we need a strategy that allows threads in each warp toaccess only one memory address to obtain elements fromthe bitmatrix in each iteration. One possible solution is toassign the same packet to w successive threads in a warpby increasing the size of a packet. This solution enables thethreads in a warp to access just one element in the bitmatrixat a time. However, it changes the access pattern to theinput data. In addition, each thread will be responsible forcoding w packets, which may reduce the opportunity forparallelism with the same workload.

Although one memory address is used to store an ele-ment in the bitmatrix (e.g., each element occupies one byte),each element can be represented by one bit because thevalue of an element is either 1 or 0. This suggests that wbits can be used to store w elements of the bitmatrix. Inpractice, the value of w can be set to no more than 8 whenk + w ≤ 256, which implies that we can use a byte to storew elements of the bitmatrix. If wm is not greater than 32,an integer can be used to carry a column vector from thebitmatrix. For this solution, the mapping for each elementfrom the bitmatrix to each memory address is illustrated inFig. 5. The access to one memory location to fetch elementsfrom the bitmatrix for a thread warp in each iteration ispresented in Fig. 6. The access latency is reduced from w

6

time slots to one time slot in each iteration. In addition,only 4kw bytes are required to store the entire bitmatrix ifmw ≤ 32. For mw greater than 32, the b32/wcw elementsof a column vector can be stored in a 32-bit integer. Theaccess pattern is similar to the case of mw ≤ 32.

x0,0 x1,0 xw-1,0…

0 1bit … w-1 …

…

mw-1

xmw-1,0

… 31Offset: 0

x0,1 x1,1 xw-1,1…

0 1bit … w-1 …

…

mw-1

xmw-1,1

… 31Offset: 1

… …

…

0 1bit … w-1 …

…

mw-1

xmw-1,kw-1

… 31Offset: kw-1

x0,kw-1 x1,kw-1 xw-1,kw-1

Fig. 5. Mapping of each bitmatrix element to memory

0 1 31tid

Iteration: i=0 j=0

0 1 … w-1Offset:

…

j=1 … j=w-1

0 1 31tid

Iteration: i=1 j=0

w w+1 … 2w-1Offset:

…

j=1 … j=w-1

… …

0 1 31tid

Iteration: i=k-1 j=0

… kw-1Offset:

…

j=1 … j=w-1

kw-w kw-w+1

Fig. 6. Memory access pattern for a single warp

After w elements are fetched from one memory address,we need to extract the exact element from the bitmatrix foreach thread. We use a variable named unit id offset tomark the index of the packet that is assigned to each threadfor each block. Using this variable, its start index in thebitmatrix can be located. When each column vector is storedin one value of int type, unit id offset can locate the bitfor its row vector elements. To detect whether the value is 1or 0, a shift operation and XOR operation is adopted. Then,the if operation is used to detect whether the XOR operationshould be performed. The problem of serialized access to thebitmatrix is completely eliminated by our proposed solution.

3.2.2 Efficient memory accessAfter removing performance penalty introduced by access-ing the bitmatrix, the major inefficient memory access comesfrom the access to the input data referenced by long *in. Fig.7 illustrates the access pattern of w successive threads toaccess wk packets from k data chunks. In the i-th iteration,the memory request is on the i-th data chunk. This accesspattern is applicable to all threads in the baseline imple-mentation.

From Table 1, we can see that shared memory is a blockof on-chip memory that can be accessed by all threads fromthe same block. This indicates that we can load data blocks

10tid

D0,0 D0,1 D0,w-1…

… w-1Iteration i=0 j=0 j=1 j=w-1

10tid

D1,0 D1,1 D1,w-1…

… w-1Iteration i=1 j=0 j=1 j=w-1

… …10tid

Dk-1,0 Dk-1,1 Dk-1,w-1…

… w-1Iteration i=k-1 j=0 j=1 j=w-1

Fig. 7. Access pattern of w threads on the data blocks

to shared memory for frequent access by threads in the sameblock. To reduce the access latency of the global memory forinput data chunks, threads in each unit cooperate with eachother to load the required data into the shared memory atthe outside iteration. The allocated shared memory for eachthread block can be reused for k times. Each thread thenaccesses the data from shared memory for calculation in theinner iteration.1 extern shared long sh data [ ] ;2 for ( i n t i = 0 ; i < k ; ++ i ) {3 i f ( threadIdx . y == 0)4 sh data [ threadIdx . x ] = ( in+ s i z e ∗ i ) + t i d ;5 synchronized ( ) ;67 for ( i n t j = 0 ; j < w; ++ j ) {8 i f ( b i t m a t r i x [ index ] == 1){9 r e s u l t ˆ= sh data [ u n i t i n i d + j ] ;

10 ++index ;11 }12 }13 synchronized ( ) ;14 }

Listing 2. Use shared memory to reduce the data access latency

The code for loading input data to shared memory isillustrated in Listing 2. The array sh data is used for storingdata from the global memory. If a thread block contains128 threads in one dimension, only 1KB of shared memoryis required for each block. Because each packet takes only8 bytes, up to 128 packets can be loaded from the globalmemory at a time. By using this optimization strategy, eachpacket of input data is accessed only once from the globalmemory.

3.2.3 Improve the utilization of memory bandwidthNow the performance of input data access can be improvedby loading data to the shared memory. However, the globalmemory bandwidth is still underutilized. When the threadswhose threadIdx.y = 0 issue an access request to the globalmemory (line 3 in Listing 2), other threads in the sameblock remain idle. Notice that the value of the variablethreadIdx.y ranges from 0 to m − 1. Because each SM hasa limited number of active blocks and active threads, thememory bandwidth is underutilized due to the inadequatenumber of memory requests to saturate the global memorybandwidth. If m is equal to 1, the memory bandwidth hasan opportunity to be fully utilized because each SM mayhave more active threads to concurrently issue memory

7

requests. However, with the increase of m, the numberof active blocks markedly decreases, implying substantialunderutilization of the global memory bandwidth due toreduced concurrent memory requests.

To conquer this issue, we assign the tasks of codingm packets to each thread to improve memory bandwidthutilization. Each thread is now responsible for coding acolumn of packets from m parity blocks. In addition toimproving the utilization of the global memory bandwidth,this approach can also remove the conditional branch at line3 in Listing 2, which leads to better performance by avoid-ing the execution of additional instructions and removingbranch divergence.

3.2.4 Optimization of control divergence

Branch divergence introduces a severe performance penaltybecause threads in a warp can execute the same instructionat a time. The execution of different instructions must beserialized for threads in a warp. In our baseline implemen-tation, branches exist in three places and may introduceperformance degradation (lines 16-18 in Listing 1).

We first investigate the if statement of the bitmatrixoperation. The purpose of this statement is to performan XOR operation with a packet where the correspondingelement from the bitmatrix is 1; otherwise, no XOR operationis performed. We use the following solution to remove thisbranch. First, we set all bits of a long type variable namedfullOnebit to 1. Then, fullOnebit performs a multiplicationwith the element extracted from the bitmatirx, denoting anoperation to copy the element from the bitmatirx (either 1or 0) to all bits of fullOnebit. Finally, fullOnebit performs anAND operation with the packet obtained from the sharedmemory. Using this approach, a branch divergence is re-moved.

For the inner loop in the baseline implementation, thesize of the loop depends on w. Since w is small in practice,we can manually apply loop unrolling to remove the innerloop for different values of w.

3.2.5 Parallelism of decoding

The encoding process can be fully parallelized because eachthread reads kw packets from the input data to code apacket. However, achieving complete parallelism may notbe straightforward for the deocding. When both a datablock and parity block become unavailable, the availablek blocks must be used together with an inverse matrix torestore the missing data block. Subsequently, w row vectorsof the generator matrix are used to restore the missing parityblock. Because all data blocks are required to serve as theinput for restoring the missing parity block, it is necessaryto wait until all data blocks become available to restore anymissing parity block. In this case, the decoding process isserialized, which reduces the decoding performance.

To reduce the performance penalty of decoding causedby this serialization, a strategy for complete parallelism ofthe decoding process must be adopted. This suggests thata matrix with mw row vectors must be used to restore anym missing blocks with any k available blocks. Moreover, fordecoding, each thread can restore a packet with input dataand a row vector. Consequently, decoding can be completely

parallelized for any case, and the decoding performance isthe same as encoding.

We first investigate how to generate a matrix that canrestore both the data blocks and parity blocks. According toFig 1, each packet from the parity blocks can be representedby the XOR results of kw packets from data blocks. We cantranslate this problem into how to represent a packet froma parity block with an XOR operation of kw input packets.Each packet can be represented by the XOR of packets fromthe kw input packets. Subsequently, the missing data packetin the representation of each missing packet from parityblocks can be replaced. After replacement and calculation,a representation of XOR operations can be obtained fromthe kw input packets. Finally, a row vector can be writtento generate the missing packets from parity blocks with kwinput packets.

We present a simple example herein to demonstrate howour proposed method works. Consider the generator matrixfrom Fig. 8 (a) as an example, where k=2, w=2, and m=2.We assume that the first data block and the first parityblock are missing. To restore packets D0,0 and D0,1, D1,0,D1,1, C1,0 and C1,1 are taken as the input. We then use theexisting application programming interface to generate tworow vectors for restoring D1,0, D1,1 as shown in Fig. 8 (a).Therefore, D0,0 can be represented as D1,1⊕C1,0⊕C1,1 andD0,1 can be represented as D1,0 ⊕ D1,1 ⊕ C1,0. Moreover,C0,0 can be represented as D0,0 ⊕ D0,1 ⊕ D1,1 and C0,1

can be represented as D0,0 ⊕ D1,0 ⊕ D1,1. By replacingD0,0 and D0,1 with their representation of input packets inthe representations of C0,0 and C0,1, we can get that C0,0

can be represented with D1,0 ⊕ D1,1 ⊕ C1,1 and C0,1 canbe represented as D1,0 ⊕ C1,0 ⊕ C1,1. Subsequently, tworow vectors can be written to restore the packets from themissing parity blocks, as illustrated in Fig .8 (b). The fourrow vectors can function with the input packets to decodethe missing blocks in parallel.

1

0 1 1 11 1 1 0

D1,0

D1,1

C1,0

C1,1

= D0,0

D0,1

(a) Recover Packets From Data Blocks

1 1 0 11 0 1 1

D1,0

D1,1

C1,0

C1,1

= C0,0

C0,1

(b) Recover Packets From Parity Blocks??

Fig. 8. An example of decoding missing blocks

We provide a function to generate the matrix that candecode the missing blocks that contain both data blocks andparity blocks. The whole matrix is placed in the bitmatrixto perform the decoding process, and the entire processbecomes the same as the encoding.

4 PERFORMANCE MODEL

The performance of a GPU application is usually boundedby either the computation capability or memory through-put. For GPU applications with low compute-to-memoryratio, the GPU’s memory bandwidth limits its performance,while the computational power limits the performance ofapplications with high compute-to-memory ratios.

Given different combinations of k, m and w, thecompute-to-memory ratio of G-CRS could be different,

8

which indicates that the performance of G-CRS may belimited to either memory bandwidth or computationalpower of a GPU. To elucidate the utilization of a GPU’scomputational resources and memory bandwidth by G-CRSunder different compute-to-memory ratios, we present aperformance model of G-CRS in this section.

4.1 Kernel Analysis

CRS coding with G-CRS includes encoding and decodingprocesses, which essentially perform the similar operations.To analyze the performance of encoding or decoding, wefirst categorize the operations in the kernel function intotwo types, namely SM and dynamic random-access memory(DRAM). SM operations include computational instruction-s, shared memory operations, constant cache operations,and synchronization instructions that are executed in SM;whereas DRAM operations include global memory read-/write transactions.

To determine the performance of G-CRS, six key opera-tions from different phases are summarized in Table 2.

TABLE 2Categorize the operation type by SM and DRAM

Phase Main operation DomainStep 3 Load data from global memory DRAMStep 3 Store data to shared memory SMStep 5 Load data from constant memory SMStep 6 Load data from shared memory SMStep 6 Computation SMStep 9 Store results to global memory DRAM

We build our G-CRS performance model based on thesmallest scheduling unit of thread warp. First, to obtainthe i-th data chunk, each warp launches a global memoryrequest to DRAM. Since each thread in a warp loads onelong integer of the i-th chunk, totally 32 long integers withlength 2048 bits are fetched from global memory. Suchmemory access should take 2048

BW transactions where BWdenotes the memory interface width of the GPU. Then thedata is stored in shared memory for later usage. Thus, k datachunks need totally 2048k

BW global memory transactions.Second, each thread loads part of the temporary data

chunk in shared memory as well as one coding bit frombitmatrix. Then they do m times of coding work to producem integers which contribute to the final m parity chunks.Each warp executes mw shared memory loading and datacoding for each data chunk. Thus, each warp executes kmwSM-type operations.

Finally, each warp writes their coding results back toglobal memory, which launches totally 2048m

BW global mem-ory transactions since each thread writes back one longinteger.

In summary, each warp has respectively 2048(k+m)BW

DRAM data transactions and kmw SM-type operations.Thus, the ratio of DRAM data transactions to SM-typeoperations varies with different values of (k,m,w), andcauses the performance to be dominated by either DRAMdata transactions or SM-type operations.

4.2 Dominant Factor Analysis

We define the memory-to-compute ratio r as Eq. (1) to betterillustrate the performance of G-CRS. Since 2048

BW is a constantthat depends on the GPU hardware, we remove it from ther definition.

r =k +m

kmw=

1/m+ 1/k

w. (1)

Each thread in a kernel issues k+m global memory requestsand conducts kmw XOR operations. Notably, r will decreasewith the increase of k,m,w. Moreover, different values of(k,m,w) lead to different memory bandwidth consump-tion, which is calculated by the total size of data divided bythe time spent on the execution. Fig. 9a presents the memorythroughput changing with r of two modern GPUs, whosephysical parameters are listed in Table 3. We define CSM byEq. (2) to describe the computational capability with respectto memory bandwidth. fcore and fmem represent the GPUcore frequency and memory frequency, respectively. #SMand #coresPerSM represent the number of SMs and thenumber of cores per SM, respectively. We have two observa-tions from Fig. 9 (a). First, memory throughput continuouslyincreases with a larger r and reaches an upper bound witha certain r. Second, two GPUs have different r thresholds,denoted by ρ, to reach the maximum memory throughput,which can be explained using CSM . A larger CSM impliesthat the hardware has a more powerful computational abili-ty, and a smaller r indicates that the computation part of thekernel scarcely limits the performance of the GPU.

CSM =fcore ×#SM ×#coresPerSM

fmem ×BW(2)

A GPU schedules its threads in rounds. In each round,the number of concurrent warps (a.k.a active warps) isdenoted by #Aw. The number of rounds is denoted by#Rs. Then the execution time of a kernel, denoted by Tkl,can be calculated as Tkl = Tact ×#Rs, where Tact denotesthe execution time of one round on one SM. Next we discusshow to calculate Tact.

As mentioned before, a kernel could be either compute-bound or memory-bound. More specifically, for our G-CRSkernel, we use ρ to help differentiate those two cases. Ifr ≤ ρ, we consider the kernel to be compute-bound andthe DRAM transaction latency can be hidden by SM-typeoperations. In this case, we only focus on the kmw SM-type operations. The execution time can be calculated bykmw × TSM × #Aw, where TSM denotes the averagenumber of cycles needed for one SM-type operation. Onthe other hand, if r > ρ, we consider the kernel to bememory-bound and the SM-type operations can be hid-den by DRAM transactions. Thus, we can focus on the2048(k+m)

BW DRAM operations, whose total time is givenby 2048(k + m)/BW × TDRAM × #Aw, where TDRAM

denotes the average number of cycles needed for one DRAMtransaction. To summarize, Tact can be calculated by Eq. 3.

We also list some key parameters of our performancemodel in Table 3. TDRAM and TSM can be obtained bymicrobenchmarks.

9

TABLE 3Metrics and parameter settings of different GPUs

Metrics/Parameter GTX980 Titan X(Pascal)Compute capability 5.2 6.1SMs * cores per SM 16 * 128 28 * 128

Memory interface width 256-bit 384-bitMemory size 4GB 12GB

Base core frequency 1126MHz 1417MHzBase memory frequency 3500MHz 5000MHz

CSM 2.57 3.21ρ 0.06 0.05

TDRAM 3.516 2.6TSM 1.506 1.5

Tact =

{kmw × TSM ×#Aw, if r ≤ ρ2048(k+m)

BW × TDRAM ×#Aw, if r > ρ(3)

To validate the accuracy of our performance model, weadopt various (k,m,w) combinations in which k rangesfrom 1 to 45, m ranges from 1 to 4 and w ranges from 4to 8, subject to the constraint of w ≥ log2(k +m). We applya large number of blocks for each kernel execution to ensurethat the GPUs keep as busy as possible. For each groupof (k,m,w), we run the kernel for ten times and calculatethe average execution time. We use mean absolute preci-sion error (MAPE) to evaluate the accuracy of our model.The MAPE is defined as MAPE = |Tmodel − Texp|/Texp,where Tmodel refers to the execution time derived from theperformance model while Texp refers to the average timemeasured by real experiments.

The accuracies of the performance estimation of thetwo GPUs are shown in Fig. 9 (b). Most of performanceestimations reveal a MAPE within 7%. The average MAPEis 2.79% for GTX980 and 3.37% for Titan X respectively. Toaddress the reproducibility of our performance model, wealso conduct significance test with bilateral t-distributionfor the average MAPE. With 95% confidence interval, ourmodel can achieve no more than 3.9% average MAPE onthese two GPUs.

0 0.1 0.2 0.3 0.4r

0

50

100

150

200

250

300

350

Mem

ory

Thr

ough

put/(

GB

/s)

gtx980Titan X(Pascal)

(a)

0.05 0.15 0.25 0.35r

0

2

4

6

8

10

Mea

n A

bsol

ute

Prec

isio

n E

rror

(%) gtx980

Titan X(Pascal)

(b)

Fig. 9. (a) Memory throughput with different r (b) Model accuracy

5 PIPELINED G-CRSFor applications with high computational intensity and alarge amount of data involved, one of the major difficultiesof obtaining high performance is to balance the cost of

computational intensity and I/O operations. For erasurecoding, we must move data to the specified GPU device andwrite the result back to main memory or network interfacethrough I/O transfer. To achieve high performance for G-CRS with both I/O and computation involved, we providea pipelined mechanism in this section that overlaps datatransfer and computation.

Efficient utilization of both the computational resourcesof GPUs and I/O bandwidth between the main memoryand GPUs’ global memory is critical for achieving peakperformance to offload the coding task to GPUs. Fig. 10(a) illustrates the execution of two tasks. Conventionally,serialized execution is adopted, and Task 2 waits until theexecution of Task 1 is completed. For pipelined execution,both the computational resources and I/O bandwidth aremore efficiently utilized. Our G-CRS resembles the execu-tion pattern of these tasks, because data must be copied frommain memory for coding and then the parity data must becopied back to the main memory.

Serialized Execution

Independent Tasks

Task 1

HtoD

Kernel

DtoH

HD1

K1

DH1

Task 2

HD2

K2

DH2

HD1

K1

DH1

HD2

K2

DH2

timePipelinedExecution

HD1

K1HD2

DH1K2

DH2

(a)

time

HD1

K1 DH1

K2

HD2

HD3

K3

DH2

DH3

HD1K1

DH1HD2

K2HD3

HD4

HD5

HD6

K3DH2

K4

K5

K6

DH3

DH4

DH5

DH6

Better UtilizationUnderutilization

(b)

Fig. 10. Cases for Pipelined I/O and Computation

To efficiently utilize both the GPU computational re-sources and I/O bandwidth, we design a pipelined mecha-nism for G-CRS to overlap I/O operations and computationtasks. We divide the coding work into independent tasksand use different GPU streams to perform each indepen-dent task in order to simultaneously run data transfer andcomputation.

Algorithm 2 illustrates how our pipeline mechanismworks. We launch a number of GPU streams to achieveasynchronous execution of each task. The execution forPipelined G-CRS is similar with the pipelined executionillustrated in Fig. 10 (a). We can observe that both GPUcomputational resources and I/O bandwidth get better uti-lization with Pipelined G-CRS by overlapping the executionof I/O tasks and computational tasks.

Algorithm 2 Pipelined G-CRS1: Input: k, w, m, bitmatrix and numStream2: for each i ∈ [0, numStream− 1] do3: Copy the i-th data chunk of stream s to GPU asyn-

chronously4: end for;5: for each i ∈ [0, numStream− 1] do6: Launch the kernel of stream i asynchronously7: end for;8: for each i ∈ [0, numStream− 1] do9: Copy back the i-th parity chunk of stream s to main

memory asynchronously10: end for;

10

TABLE 4Major Parameters for Measuring Different Workloads

Parameter Value Rangek 10m 1-4w 4-8

Thread Blocks 24-213

Threads 128 Per Thread BlockInput data size 8kwb128/wc Bytes Per Thread Block

An example of executing three independent coding tasksis illustrated on the left side of Fig. 10 (b). The total executiontime is 3tHD + tDH + tK , which reveals that the time forexecuting all tasks comprises three parts. The first part is thetime required to transfer all data to the GPU, the second partis the time required for the kernel execution, and the thirdpart is the time required to transfer the data from the GPUback to the main memory. Overall, dividing each task intosmaller tasks can reduce the total execution time. The rightside in Fig. 10 (b) depicts the execution of each independenttask divided into two equal sized tasks, for which the totalexecution time is reduced to 3tHD + tDH

2 + tK2 . Assuming

that the total time for transferring all of the data from mainmemory to GPU memory is THD , the total time for codingis TK , and the total time for transferring the entire paritychunks from GPU memory to main memory is TDH . Wedivide the work into x tasks so that each task requires THD

x

time to transfer its data to the GPU memory, TK

x to producethe parity chunks, and TDH

x to transfer the parity chunksback to the main memory. The total time for completing theentire task is calculated as follows:

Ttotal = THD +TKx

+TDH

x(4)

Equation (4) reveals that the entire coding task can bedivided into many smaller tasks that can effectively utilizeboth the I/O bandwidth and computation resources toachieve better performance.

6 PERFORMANCE EVALUATION

In this section, we conduct a set of experiments toevaluate G-CRS from various aspects on two modern GPUarchitectures: Maxwell and Pascal. First, we evaluate thecoding performance of G-CRS with various workloads. Nex-t, we analyze the peak raw coding performance of G-CRSand compare it with the performance of other major state-of-the-art coding libraries: Jerasure, Gibraltar, gferasure [39]and PErasure [17]. We also analyze how our different opti-mization strategies improve the performance of CRS coding.

6.1 Throughput Under Different Workloads

The coding performance of G-CRS depends on the levelof parallelism on a given GPU for the specified k, m, andw values. The level of parallelism that can be achieveddepends on the workload assigned to each task. In otherwords, an adequate workload must be assigned to a givenGPU to achieve a high throughput.

We measure the coding performance of G-CRS underdifferent workloads on two modern generations of GPUs:Maxwell and Pascal. The major parameters for our exper-iments are presented in Table 4. First, we set the numberof threads in each thread block to 128 because this yieldsthe maximum number of active threads and the maximumnumber of thread thread blocks in each SM for Maxwell andPascal. Notably, the input data size is slightly different fordifferent w values due to memory alignment. The numberof working threads in each thread blocks is w b128/wc, andthe input data size is the total number of thread blocksmultiplied with 8kw b128/wc bytes.

The experimental results are presented in Fig. 11. Con-sidering the coding performance with minimum threadblocks for a kernel, the throughput of the Maxwell GTX 980is around 11 GB/s when m equals 1. We can see that theperformance is doubled when the workload is doubled form = 1 or 2. But for the cases of m = 3 or 4, there is nobig performance improvement. This is because larger valueof m requires more computation resources. The situationis similar for Pascal Titan X GPU. But it has a higherthroughput with minimum number of chunks for a kernel,because it has higher memory bandwidth than MaxwellGTX 980.

At maximum workload, in which each kernel has 213

thread blocks, the throughput reaches the peak for eachGPU. Moreover, when computational power is not the bot-tleneck of the coding performance, the peak throughputis limited by the GPU memory bandwidth; therefore, thethroughput of the Maxwell GTX 980 is about half of thatof the Pascal Titan X when m = 1. However, when eachthread is responsible for coding more packets, the compu-tational power is gradually saturated and hence limits thecoding throughput. With maximum workload, the codingthroughput decreases to around two third of its originalpeak when m is increased from 1 to 4. For Pascal Titan X,which has a higher memory bandwidth and computationalpower, the decrease is not so obvious as the Maxwell GTX980. Specifically, the peak throughput of the Pascal Titan X isapproximately 230 GB/s when each thread is assigned withfour packets.

The value of w also has a great impact on the codingperformance. Specifically, a GPU performs optimally whenit has the smallest value of w, which is 4, than with anyother values. Especially when computation is the limitation,smaller w leads to better performance because the increasein w indicates an increase in computational intensity foreach working thread. Furthermore, when bandwidth is thelimitation, w with values of 4 and 8 may outperform w withother values because of the increased number of workingthreads in each thread block; thus, a more efficient band-width utilization is achieved. Based on our experimentalresults, we conclude that when the computation is the bot-tleneck, w with a smaller value outperforms w with largervalues. On the other hand, when GPU memory bandwidthis the bottleneck, more working threads in each thread blockcan lead to better performance.

We present only the encoding performance to representthe overall coding performance of G-CRS herein because thedifference between encoding and decoding throughput isless than 1/1,000 in our measurement. The evaluation result-

11

1

24 25 26 27 28 29 2102112122130

153045607590

105120135150

Thread Blocks For a Kernel

Thro

ughp

ut(G

B/s)

Maxwell GTX 980

(a) m=1

24 25 26 27 28 29 2102112122130

13263952657891

104117130

Thread Blocks For a Kernel

Thro

ughp

ut(G

B/s)

Maxwell GTX 980

(b) m=2

24 25 26 27 28 29 2102112122130

1224364860728496

108120

Thread Blocks For a Kernel

Thro

ughp

ut(G

B/s)

Maxwell GTX 980

(c) m=3

24 25 26 27 28 29 2102112122130

112233445566778899

110

Thread Blocks For a Kernel

Thro

ughp

ut(G

B/s)

Maxwell GTX 980

(d) m=4

24 25 26 27 28 29 2102112122130

295887

116145174203232261290

Thread Blocks For a Kernel

Thro

ughp

ut(G

B/s)

Pascal Titan X

(e) m=1

24 25 26 27 28 29 2102112122130

20406080

100120140160180200220240260

Thread Blocks For a Kernel

Thro

ughp

ut(G

B/s)

Pascal Titan X

(f) m=2

24 25 26 27 28 29 2102112122130

255075

100125150175200225250

Thread Blocks For a Kernel

Thro

ughp

ut(G

B/s)

Pascal Titan X

(g) m=3

24 25 26 27 28 29 2102112122130

20406080

100120140160180200220240

Thread Blocks For a Kernel

Thro

ughp

ut(G

B/s)

Pascal Titan X

(h) m=4w=4 w=5 w=6 w=7 w=8

Fig. 11. Throughput under Different Workloads

1

0 10 20 30 40 500

285684

112140168196224252280

Value of k

Thr

ough

put(

GB/

s)

(a) m=2

0 10 20 30 40 500

255075

100125150175200225250

Value of k

Thro

ughp

ut(G

B/s)

(b) m=4

0 10 20 30 40 500

1224364860728496

108120

Value of k

Thro

ughp

ut(G

B/s)

(c) m=8

0 10 20 30 40 5006

121824303642485460

Value of k

Thro

ughp

ut(G

B/s)

(d) m=16G-CRS 980 G-CRS Titan X PErasure 980 PErasure Titan X Gibraltar 980

Gibraltar Titan X gferasure AVX2 gferasure SSSE3 JerasureMulti JerasureSIMDMulti

Fig. 12. Comparison of raw encoding performance

s also reveal that more workloads typically indicate a moreefficient coding performance because of higher parallelism.At minimum workload, the throughput is usually sufficientto saturate 40G/100G-bps of Ethernet.

6.2 Peak Raw Coding PerformanceAfter investigating the coding performance under differentworkloads, we measure the peak raw coding performanceof our G-CRS and compare it with other state-of-the-artcoding libraries, namely Jerasure, Gibraltar, gferasure andPErasure. Jerasure is a well-known erasure coding libraryon CPU [40]. We measure two versions of Jerasure: one withCRS coding and one with accelerated Reed-Solomon codingthat uses Intel SIMD instructions presented in [30]. Gibraltaris a recently developed open-source Reed-Solomon codinglibrary for GPUs. gferasure is an on-going erasure codelibrary accelerated with a high-performance Galois fieldarithmetic [39]; we measure its performance using SSSE3and AVX2, which achieve the fastest coding performance ofgferasure.

The CPU used for evaluating Jerasure and gferasure isan Intel i7-6800K model with six cores running at 3.4GHz.To maximize the coding performance of the CPU, we usedOpenMP to parallel Jerasure and gferasure. We set thevalues of m to 2, 4 and 8, and the values of k range fromm to 45. For Jerasure with CRS coding, we set the packetsize to 8 KB to achieve maximum coding performance. Thesize of each input data chunk is fixed to 10 MB. Finally, weselected the smallest available value of w for erasure coding.Notably, the size of each data chunk varies slightly differentdue to the necessary alignment for w with values of 5, 6 and7.

The encoding performance is presented in Fig. 12. Asdepicted, our G-CRS is more than ten times faster than theGibraltar, Jerasure and gferasure, and up to 3 times fasterthan PErasure on the same architecture.

Jerasure shows the lowest coding throughput in all ex-periments. Although its Reed-Solomon coding outperformsCRS coding with a single thread, the CRS coding achieves3.5x higher performance with four threads, but the Reed-

12

Solomon coding only achieves 1.5x higher performancewhen using four threads. By contrast, our G-CRS outper-forms Jerasure by more than 10x in all situations.

Gibraltar achieves a coding throughput of approximately11 GB/s for m with values of 2 and 4. The performance ofGibraltar is limited by the PCIe bandwidth because it useszero copy to manage the memory buffer, which indicatesthat it accesses the host memory directly for coding. How-ever, when m is equal to 8, the coding throughput was only8 GB/s on the Maxwell GTX 980, where the PCIe bandwidthno longer limited its performance.

The coding performance of gferasure is markedly faston the CPU. In addition, gferasure with AVX2 outperformsthat with SSSE3, because AVX2 can process a 256-bit wordwhich is double of the length of SSSE3. We also achievesa 2x speedup with multi-threads in our measurement. Italso outperforms Gibraltar when m = 2 and 4. Still, ourG-CRS is five to ten times faster than gferasure because G-CRS exhibits a better utilization of GPUs’s computationalpower and memory bandwidth.

In short, G-CRS can effectively exploit GPU’s compu-tational power and memory bandwidth, and exhibits amore efficient coding performance compared with otherstate-of-the-art coding libraries. In all cases, G-CRS’s codingperformance can saturate the state-of-the-art disk I/O andnetwork bandwidth.

6.3 Optimization Analysis

We have proposed a set of optimization strategies to over-come every possible performance penalty to build G-CRS.In this subsection, we evaluate the effectiveness of ouroptimization strategies and then investigate the dominatingfactor that contributes the most to the performance improve-ment.

6.3.1 Effectiveness of optimization strategiesWe evaluate the effectiveness of our optimization strategiesby gradually applying them to the baseline version in List-ing 1. We fix k as 10 and set m in the range of 1 to 4. The valueof w varies from 4 to 8 for different computation workloads.The size of each data chunk is set at 10 MB to achieve thepeak coding performance.

The evaluation results are presented in Fig. 13, whereinonly the encoding performance is shown to represent theentire coding performance. The Base version is the base-line implementation presented in Listing 1. The Bitmatrixversion stores each column vector from the bitmatrix inone memory address. The one named EfficientAccess isthe version that we apply shared memory optimizationin Listing 2 on the Bitmatrix version. For the Bandwidthversion, we reduce the thread dimension of y to 1, and eachworking thread is assigned with m packets. G-CRS is ourfinal version with all optimization strategies applied.

It is obvious that the Base version has the lowest codingperformance, about 10% of the highly optimized G-CRS.When we change the way how bitmatrix is stored andaccessed, we observe a big performance improvement fromthe Base version to Bitmatrix version. But the degree ofimprovement decreases greatly when m is increasing. Thisis because more working threads are launched with the

increase of m, which increases the access to data chunks andbrings more performance penalties.

We can see that the version named EfficientAccess hasmuch better performance than the Bitmatrix when m =2, 3, 4, and the one named Bandwidth further improvesthe performance of EfficientAccess. Our G-CRS obtains thebest coding performance in all experiments. The one namedBandwidth has the closest performance to our G-CRS. Fromthis we first can see that the better utilization of bandwidthcan bring better performance. And we also can see thatthe control divergence brings the least performance penaltycompared with others.

We can see an obvious performance improvement afteradding our optimization strategy one by one to form G-CRS. Thus, we can conclude that each of our optimization iseffective to improve the coding performance.

6.3.2 Dominating factorWe have shown that our optimization strategies can im-prove the performance of the Base version greatly. To findout the dominating factor for the performance improvemen-t, we remove one optimization strategy from the G-CRSand keep the remaining ones to observe its impact on theperformance.

Fig. 14 illustrates the experimental results. We use theminus symbol to represent that the corresponding optimiza-tion strategy is not used. For example, -Bitmatrix means weremove the optimization of bitmatrix from G-CRS.

It is obvious to see that without the optimization of bit-matrix, the performance degrades significantly. Even thoughother optimizations are used, its performance is only slightlybetter than the Base version in Fig. 13. There are twoperformance penalties for accessing bitmatrix from globalmemory directly. One is the long access latency to fetch anelement from global memory. Another one is the serializedaccess to different elements due to non-coalescing access.From our experimental results, we can conclude that theoptimization of the bitmatrix is the dominating factor to theperformance improvement.

For others with the optimization of bitmatrix, we can seethe performance penalty is reduced. Without loading datafrom global to on-chip shared memory, which is the versionof -EfficientAccess, the performance decreases obviouslycompared with G-CRS.

The performance of the -Bandwidth is similar with G-CRS when m is 1. But when m increases, there is a per-formance gap between -Bandwidth and G-CRS becausethe memory bandwidth is underutilized with the increaseof m in the version of -Bandwidth. The version of -ControlDivergenceOpt is the one that is closest to the per-formance of our G-CRS. However, the performance gapincreases with the increase of m and w.

In a word, the optimization to bitmatrix is the dominatingstrategy that can improve the performance by 4-7 times,while other optimization strategies can further improve theperformance by 30-100%.

6.4 Overall Performance

For both encoding and decoding, data must be transferredfrom other sources to the device memory via PCIe. We

13

1

3 4 5 6 7 8 90

153045607590

105120135150

Value of w

Thr

ough

put(

GB/

s)

Maxwell GTX 980

(a) k=10, m=1

3 4 5 6 7 8 90

14284256708498

112126140

Value of w

Thr

ough

put(

GB/

s)

Maxwell GTX 980

(b) k=10, m=2

3 4 5 6 7 8 90

1224364860728496

108120

Value of w

Thr

ough

put(

GB/

s)

Maxwell GTX 980

(c) k=10, m=3

3 4 5 6 7 8 90

13263952657891

104117

Value of w

Thr

ough

put(

GB/

s)

Maxwell GTX 980

(d) k=10, m=4

3 4 5 6 7 8 90

295887

116145174203232261290

Value of w

Thr

ough

put(

GB/

s)

Pascal Titan X

(e) k=10, m=1

3 4 5 6 7 8 90

275481

108135162189216243270

Value of w

Thr

ough

put(

GB/

s)

Pascal Titan X

(f) k=10, m=2

3 4 5 6 7 8 90

255075

100125150175200225250

Value of w

Thro

ughp

ut(G

B/s)

Pascal Titan X

(g) k=10, m=3

3 4 5 6 7 8 90

24487296

120144168192216240

Value of w

Thro

ughp

ut(G

B/s)

Pascal Titan X

(h) k=10, m=4Base Bitmatrix EfficientAccess Bandwidth G-CRS

Fig. 13. Optimization analysis

1

3 4 5 6 7 8 90

153045607590

105120135150

Value of w

Thro

ughp

ut(G

B/s)

Maxwell GTX 980

(a) k=10, m=1

3 4 5 6 7 8 90

14284256708498

112126140

Value of w

Thro

ughp

ut(G

B/s)

Maxwell GTX 980

(b) k=10, m=2

3 4 5 6 7 8 90

13263952657891

104117130

Value of w

Thro

ughp

ut(G

B/s)

Maxwell GTX 980

(c) k=10, m=3

3 4 5 6 7 8 90

1224364860728496

108120

Value of w

Thro

ughp

ut(G

B/s)

Maxwell GTX 980

(d) k=10, m=4

3 4 5 6 7 8 90

306090

120150180210240270300

Value of w

Thro

ughp

ut(G

B/s)

Pascal Titan X

(e) k=10, m=1

3 4 5 6 7 8 90

275481

108135162189216243270

Value of w

Thro

ughp

ut(G

B/s)

Pascal Titan X

(f) k=10, m=2

3 4 5 6 7 8 90

255075

100125150175200225250

Value of w

Thro

ughp

ut(G

B/s)

Pascal Titan X

(g) k=10, m=3

3 4 5 6 7 8 90

24487296

120144168192216240

Value of w

Thro

ughp

ut(G

B/s)

Pascal Titan X

(h) k=10, m=4-Bitmatrix -EfficientAccess -Bandwidth -ControlDivergenceOpt G-CRS

Fig. 14. Dominating factor analysis

14

evaluate the coding performance of our pipelined G-CRSwith PCIe transfer involved. The PCIe used in our ex-periment is PCIe 3.0, which supports bi-directional PCIecommunications. Our benchmarking experiments show thatthe highest effective PCIe bandwidth on our testbed is 12GB/s. We set each data chunk size to 10 MB, divide theentire coding task into 10 small tasks, and use 10 CUDAstreams to implement pipelined G-CRS.

The encoding performance is presented in Fig. 15. Sinceno obvious differences have been observed between encod-ing and decoding performance, the decoding performance isnot presented. By using pipelined G-CRS, the coding speedreaches up to 11 GB/s, which is approximately 90% of thebandwidth limitation. This overall coding throughput canalready saturate the network bandwidth. With the recentprogress on the standardization of PCIe 4.0 and PCIe 5.0[41], the overall performance of G-CRS can be naturallyboosted in the future.

7 CONCLUSION

In this paper, we presented the design of a GPU acceleratedCRS coding and evaluated its performance on two currentgenerations of GPUs. Our evaluation results indicate thatG-CRS can operate with various sizes of data and codeblock to obtain a favorable coding throughput. The cod-ing performance of G-CRS can be up to ten times fasterthan other state-of-the-art coding libraries such as Jerasure,Gibraltar and gferasure. Moreover, our optimization strate-gies can effectively utilize both GPU memory bandwidthand computation resources. The overall performance of G-CRS is only limited by PCIe bandwidth. Our optimizationstrategies can also be applied to other applications withsimilar compute and data access patterns. Examples includeregenerating codes that are built on top of Cauchy Reed-Solomon codes [42], and also circular-shift linear networkcoding [43].

ACKNOWLEDGEMENT

We would like to thank the anonymous reviewers for theirvaluable comments. We gratefully acknowledge the supportof NVIDIA Corporation with the donation of the Titan XPascal GPU used for this research. This work is partiallysupported by Shenzhen Basic Research Grant SCI-2015-SZTIC-002 and Hong Kong ITF Grant ITS/443/16FX.

REFERENCES

[1] M. Grawinkel, Nagel et al., “Analysis of the ECMWF storagelandscape,” in Proc. of USENIX FAST’15, 2015.

[2] C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li,and S. Yekhanin, “Erasure coding in Windows Azure Storage,” inProc. of USENIX ATC’12, 2012.

[3] K. Rashmi, N. B. Shah et al., “A solution to the network challengesof data recovery in erasure-coded distributed storage systems: Astudy on the facebook warehouse cluster,” 2013.

[4] S. Muralidhar, W. Lloyd et al., “F4: Facebook’s warm blob storagesystem,” in Proc. of USENIX SOSP’14, 2014.

[5] B. Fan, W. Tantisiriroj et al., “Diskreduce: Replication as a preludeto erasure coding in data-intensive scalable computing,” ParallelData Laboratory, Carnegie Mellon University, Pittsburgh, 2011.

[6] S. A. Weil, S. A. Brandt et al., “Ceph: A scalable, high-performancedistributed file system,” in Proc. of USENIX OSDI’06, 2006.

[7] M. Blaum and R. M. Roth, “On lowest density MDS codes,” IEEETransactions on Information Theory, vol. 45, no. 1, pp. 46–59, 1999.

[8] I. Reed and G. Solomon, “Polynomial codes over certain finitefields,” Journal of the Society for Industrial and Applied Mathematics,vol. 8, no. 2, pp. 300–304, 1960.

[9] J. Bloemer, M. Kalfane et al., “An XOR-based erasure-resilientcoding scheme,” ICSI Technical Report No. TR-95-048, 1995.

[10] C. Binnig, A. Crotty et al., “The end of slow networks: It’s time fora redesign,” vol. 9, no. 7, 2016, pp. 528–539.

[11] J. Meza, Q. Wu, S. Kumar, and O. Mutlu, “A large-scale study offlash memory failures in the field,” in Proc. of ACM SIGMETRIC-S’15, 2015.

[12] P. Bhatotia, R. Rodrigues, and A. Verma, “Shredder: GPU-accelerated incremental storage and computation.” in Proc. ofUSENIX FAST’12, 2012.

[13] W. Sun, R. Ricci, and M. L. Curry, “GPUstore: Harnessing GPUcomputing for storage systems in the OS kernel,” in Proc. of ACMSYSTOR’12, 2012.

[14] A. Khasymski, M. M. Rafique et al., “On the use of GPUs inrealizing cost-effective distributed RAID,” in Proc. of IEEE MAS-COTS’12, 2012.

[15] M. L. Curry, A. Skjellum et al., “Gibraltar: A Reed-Solomon codinglibrary for storage applications on programmable graphics proces-sors,” Concurrency and Computation: Practice and Experience, vol. 23,no. 18, pp. 2477–2495, 2011.

[16] J. S. Plank, S. Simmerman, and C. D. Schuman, “Jerasure: A li-brary in c/c++ facilitating erasure coding for storage applications-version 1.2,” Citeseer, Tech. Rep., 2008.

[17] X. Chu, C. Liu et al., “Perasure: a parallel cauchy reed-solomoncoding library for GPUs,” in Proc. of IEEE ICC’15, 2015.

[18] X. Mei and X. Chu, “Dissecting GPU memory hierarchy throughmicrobenchmarking,” IEEE Transactions on Parallel and DistributedSystems, vol. 28, no. 1, pp. 72–86, 2017.

[19] N. Corporation, “NVIDIA CUDA programming guide,” http://docs.nvidia.com/cuda/cuda-c-programming-guide.

[20] W. Lin, D. M. Chiu, and Y. Lee, “Erasure code replication revisit-ed,” in Peer-to-Peer Computing, 2004, pp. 90–97.

[21] B. Jasmeet, M. Hany, and Y. Zhiping, “Opening designs for 6-packand wedge 100,” https://code.facebook.com/posts/203733993317833/opening-designs-for-6-pack-and-wedge-100/ .

[22] M. Yuval Bachar, “The linkedin data center 100gtransformation,” https://engineering.linkedin.com/blog/2016/03/the-linkedin-data-center-100g-transformation.

[23] W. Wang, T. Xie, and A. Sharma, “SWANS: An interdisk wear-leveling strategy for RAID-0 structured SSD arrays,” Trans. Storage,vol. 12, no. 3, Apr. 2016.

[24] J. S. Plank and L. Xu, “Optimizing cauchy reed-solomon codesfor fault-tolerant network storage applications,” in Proc. of IEEENCA’06, 2006.

[25] J. Luo, M. Shrestha, L. Xu, and J. S. Plank, “Efficient encodingschedules for XOR-based erasure codes,” IEEE Transactions onComputers, vol. 63, no. 9, pp. 2259–2272, 2014.

[26] H. Klein and J. Keller, “Storage architecture with integrity, redun-dancy and encryption,” in Proc. of IEEE IPDPS’09, 2009.

[27] P. Sobe, “Parallel reed/solomon coding on multicore processors,”in Proc. of IEEE SNAPI’10, 2010.

[28] J. Feng, Y. Chen, and D. Summerville, “EEO: An efficient MDS-likeRAID-6 code for parallel implementation,” in Proc. of IEEE SarnoffSymposium, 2010.

[29] J. Feng, Y. Chen et al., “An extension of RDP code with paralleldecoding procedure,” in Proc. of IEEE CCNC’12, 2012, pp. 154–158.

[30] J. S. Plank, K. M. Greenan, and E. L. Miller, “Screaming fast galoisfield arithmetic using intel SIMD instructions.” in Proc. of USENIXFAST’13, 2013.

[31] K. Zhao and X. Chu, “G-BLASTN: Accelerating nucleotide align-ment by graphics processors,” Bioinformatics, vol. 30, no. 10, pp.1384–1391, 2014.

[32] G. Vasiliadis, M. Polychronakis et al., “Regular expression match-ing on graphics hardware for intrusion detection,” in Proc. ofInternational Workshop on Recent Advances in Intrusion Detection,2009.

[33] K. Jang, S. Han et al., “SSLShader: Cheap SSL acceleration withcommodity processors.” in Proc. of USENIX NSDI’11, 2011.

[34] Q. Li, C. Zhong et al., “Implementation and analysis of AESencryption on GPU,” in Proc. of IEEE HPCC-ICESS, 2012.

[35] X. Chu, K. Zhao, and M. Wang, “Practical random linear networkcoding on GPUs,” in Proc. of IFIP Networking’09, 2009.

15

1

0 10 20 30 40 500123456789

101112

Value of k

Thro

ughp

ut(G

B/s)

(a) m=2

0 10 20 30 40 500123456789

101112

Value of k

Thr

ough

put(

GB/

s)

(b) m=4

0 10 20 30 40 500123456789

101112

Value of k

Thro

ughp

ut(G

B/s)

(c) m=8

0 10 20 30 40 500123456789

101112

Value of k

Thr

ough

put(

GB/

s)

(d) m=16Bandwidth Limit G-CRS 980 G-CRS Titan X

Fig. 15. Overall encoding performance

[36] X. Chu and Y. Jiang, “Random linear network coding for peer-to-peer applications,” IEEE Network, vol. 24, no. 4, pp. 35–39, 2010.

[37] H. Shojania, B. Li, and X. Wang, “Nuclei: GPU-accelerated many-core network coding,” in Proc. of IEEE INFOCOM’09, 2009.

[38] N. Corporation, “Kepler tuning guide,” http://docs.nvidia.com/cuda/kepler-tuning-guide.

[39] L. T. Ethan Miller, Thomas Schwarz and A. Kwong, “High-performance galois field arithmetic,” http://www.crss.ucsc.edu/proj/galois.html.

[40] J. S. Plank, J. Luo et al., “A performance evaluation and examina-tion of open-source erasure coding libraries for storage.” in Proc.of USENIX FAST’09, 2009.

[41] E. Born, “Pcie 4.0 specification finally out with16 gt/s on tap,” https:// techreport.com/news/32064/pcie-4-0-specification-finally-out-with-16-gt-s-on-tap, 2017.

[42] K. Rashmi, N. B. Shah et al., “A hitchhiker’s guide to fast andefficient data reconstruction in erasure-coded data centers,” inProc. of ACM SIGCOMM’14, 2014.

[43] H. Tang, Q. T. Sun, Z. Li, X. Yang, and K. Long, “Circular-shiftlinear network coding,” arXiv preprint arXiv:1707.02163, 2017.

Chengjian Liu received his MS degree in Col-lege of Computer Science & Software Engineer-ing, SZU. He has been working in GuangdongProvince Key Laboratory of Popular High Per-formance Computers & Shenzhen Key Labo-ratory of Service Computing and ApplicationsHigh Performance Computing during his study inSZU. Currently he is a Ph.D. candidate at Deptof Computer Science, Hong Kong Baptist Uni-versity. His research interests include DistributedStorage, General-Purpose GPU Computing.

Qiang Wang received his B.Sc. degree fromSouth China University of Technology in 2014.He is currently a Ph.D. candidate at Departmentof Computer Science, Hong Kong Baptist Uni-versity. His research interests include General-Purpose GPU Computing and power-efficientcomputing. He is a recipient of Hong Kong PhDFellowship.

Xiaowen Chu received the B.E. degree in com-puter science from Tsinghua University, P.R. Chi-na, in 1999, and the Ph.D. degree in comput-er science from the Hong Kong University ofScience and Technology in 2003. Currently, heis an associate professor in the Department ofComputer Science, Hong Kong Baptist Universi-ty. His research interests include distributed andparallel computing and wireless networks. He isserving as an Associate Editor of IEEE Accessand IEEE Internet of Things Journal. He is a

senior member of the IEEE.

Yiu-Wing Leung received his B.Sc. and Ph.D.degrees from the Chinese University of HongKong. He has been working in the Hong KongBaptist University and now he is Professorof the Computer Science Department andProgramme Director of two MSc programmes.His research interests include three major areas:1) network design, analysis and optimization, 2)Internet and cloud computing, and 3) systemsengineering and optimization. He has publishedmany papers in various IEEE journals and

conferences.