QGAN: Quantized Generative Adversarial Networks

Peiqi Wang 1 Dongsheng Wang 1 Yu Ji 1 Xinfeng Xie 2 Haoxuan Song 1

Xuxin Liu 1 Yongqiang Lyu 1 Yuan Xie 2

1Department of Computer Science and Technology, Tsinghua University2Department of Electrical and Computer Engineering, University of California, Santa Barbara

wpq14@mails.tsinghua.edu.cn, wds@mail.tsinghua.edu.cn

AbstractThe intensive computation and memory require-ments of generative adversarial neural networks(GANs) hinder its real-world deployment on edgedevices such as smartphones. Despite the suc-cess in model reduction of CNNs, neural networkquantization methods have not yet been studied onGANs, which are mainly faced with the issues ofboth the effectiveness of quantization algorithmsand the instability of training GAN models. In thispaper, we start with an extensive study on apply-ing existing successful methods to quantize GANs.Our observation reveals that none of them gen-erates samples with reasonable quality becauseof the underrepresentation of quantized values inmodel weights, and the generator and discrimi-nator networks show different sensitivities uponquantization methods. Motivated by these obser-vations, we develop a novel quantization methodfor GANs based on EM algorithms, named asQGAN. We also propose a multi-precision algo-rithm to help find the optimal number of bits ofquantized GAN models in conjunction with corre-sponding result qualities. Experiments on CIFAR-10 and CelebA show that QGAN can quantizeGANs to even 1-bit or 2-bit representations withresults of quality comparable to original models.

1. IntroductionGenerative adversarial networks (GANs) have obtained im-pressive success in a wide range of applications, such assuper-resolution image generation, speech synthesis, image-to-image translation, and video frame prediction (Bulat et al.,2018; Ao et al., 2018). Despite their success in generatinghigh-quality samples, these models are hard to be deployedinto real-world applications on edge devices because of theirhuge demands for computation and storage capacity. For ex-ample, the BigGAN model (Brock et al., 2018), developedby Google, contains up to 0.2 TOPs in the inference phase

and its model size is over 1.3 GB.This challenge becomesmore urgent as the growth of privacy and security concernsabout running the inference on cloud platforms.

This challenge exists in the deployment of various neuralnetwork models besides GANs. State-of-the-art techniquesto compress model scales include pruning, quantizaion, low-rank approximate (Han et al., 2015; Courbariaux et al.,2016; Sainath et al., 2013). Among these techniques, quan-tization is the most easy-to-use and scalable method, whichuses a less number of bits for data representations than a32-bit single-precision. Quantization has the following threeaspects of advantages over other techniques. First, the com-pression rate is significant. For example, the model size ofa 2-bit quantized model is reduced by 16×. Second, thequantization technique does not change neural network ar-chitectures. Thus it is orthogonal to the study of algorithmsfor neural network architecture exploration. Finally, it iseasy to be deployed into off-the-shelf devices with littlehardware co-design to obtain significant performance andenergy benefits. The use of quantization methods requireslittle knowledge from algorithm researchers to hardware.

Although quantizing neural network models has achievedimpressive success on convolutional neural networks(CNNs) and recurrent neural networks (RNNs) (Zhou et al.,2016; Rastegari et al., 2016), there is still no successful at-tempt to quantize GAN models. In this paper, we first studythe effectiveness of typical quantization methods on GANmodels. Despite the success of these methods on CNNs orRNNs, We observe that they are not directly applicable toquantize GAN models because of the underrepresentationto original values. Besides, we observe features of the con-vergence and sensitivity of quantized GAN model. Basedon our observations, we develop QGAN, a novel quanti-zation method based on Expectation-Maximization (EM)algorithm, and a novel multi-precision quantization algo-rithm. Finally, our experiments show that the proposed Qcan compress GAN models into 1-bit or 2-bit representa-tions while generating samples of comparable quality, andour multi-precision method helps further improve the re-sults quality of quantized GAN models according to a given

arX

iv:1

901.

0826

3v1

[cs

.NE

] 2

4 Ja

n 20

19

QGAN: Quantized Generative Adversarial Networks

demand.

In summary, our work has following contributions:

• We conduct an extensive study on the effectivenessof existing widely-used quantization methods. Thisempirical study demonstrates that these quantizationmethods are not applicable to GAN models directlyalthough they work well on CNNs or RNNs.

• We obtain some observations of quantized GAN. First,the discriminator is more sensitive than the generatorto the number of quantized bits. Second, a convergedquantized discriminator can ensure the convergence ofthe entire quantized GAN model. Third, quantizingboth the discriminator and generator is more stablethan only quantizing generator networks.

• We propose QGAN, a novel quantization method forGAN models based on EM algorithm to overcome thedata underrepresentation problem of existing quantiza-tion methods. Our experiments demonstrate that GANmodels quantized to 2-bit or even 1-bit by QGAN cangenerate samples of comparable quality.

• We develop a multi-precision quantization algorithmbased on our observations on the convergence and sen-sitivity of quantized GAN models. This algorithmprovides the optimal number of bits configuration tosatisfy the results quality requirement.

2. Background2.1. Generative adversarial networks

Generative Adversarial Network (GAN) is composed oftwo components, the generator and the discriminator. Thegenerator network, usually denoted as G is trained to gener-ate samples in a similar distribution of real data while thediscriminator network, usually denoted as D is trained todiscriminate whether the input is generated by G or fromreal data. The generator takes a sampled noise z, wherez ∼ N (0, 1) or U(−1, 1), as the input each time to gener-ate a data sample. Both samples generated from G and realdata are taken as inputs, denoted as x, to the discriminator,and the discriminator estimate the probability, D(x), thatthe input is from real data. The training process of a GANmodel can be formulated as a min-max game between thegenerator and the discriminator. The objective function ofthis min-max game can be formulated as:

minG

maxD

V (D,G) =Ex∼pdata(x)[logD(x)]+

Ez∼p(z)[log(1−G(z)](1)

The generator aims to minimize this objective function whilethe discriminator aims to maximize it. Both of them con-

verge at a Nash equilibrium point where neither of them hasany better action to further improve objects.

To improve the quality of generated samples, prior stud-ies focus on better neural network architectures (Radfordet al., 2015; Mirza & Osindero, 2014; Karras et al., 2017).Some studies propose new objective functions for betterconvergence of the training process, such as adding newconstraints (Arjovsky et al., 2017; Gulrajani et al., 2017)and using smoother non-vanishing or non-exploding gradi-ents (Mao et al., 2017; Zhao et al., 2016). Our work focuseson using a smaller number of bits for data representationsin GAN models for a more efficient deployment on edgedevices, thus our work is orthogonal to these prior studies.

2.2. Quantizaion

Quantization is a promising technique to reduce neural net-work model size and simplify arithmetic operations by re-ducing the number of bits in the data representation. For ex-ample, in binary neural networks (Courbariaux et al., 2016),both weights and activations are quantized to 1-bit fromoriginal full-precision representation (32-bit). In this case,the size of the model is reduced by 32× and the floating-point arithmetic operations are simplified into single-bitlogical operations (Rastegari et al., 2016). From the per-spective of hardware, these operations are easier to be imple-mented with higher performance, better energy efficiency,and smaller area overheads. Therefore, these quantizedmodels are easier to be deployed on edge devices becauseof smaller model sizes and hardware-friendly operations.

The benefits of quantization motivate prior research studiesin CNNs and RNNs. Among these studies, Binarized Neu-ral Network (Courbariaux et al., 2016) uses a single signfunction with a scaling factor to binarize the weights and ac-tivations. XNOR-Net (Rastegari et al., 2016) formulates thequantization as an optimization problem, and successfullyquantize CNNs to a binary neural network without accuracyloss. DoReFa-Net (Zhou et al., 2016) adopts heuristic linearquantization to weights, activations, and gradients. Somestudies pay attention to quantizing networks to extreme lowbits (Courbariaux et al., 2016; Zhu et al., 2016; Wang et al.,2018), and some focus on quantizing more objects (Mi-cikevicius et al., 2017; Banner et al., 2018), e.g. gradients,errors, and weight update. These studies have achievedgreat success by using an impressive low number of bits,usually 1-bit or 2-bit, while obtaining accuracy comparableto full-precision baseline models.

Despite these successful results on CNN and RNN, our workis the first to focus on quantizing GAN models. Our casestudy in Section 3.2 shows that these typical methods usedin CNN and RNN models are not directly applicable to quan-tize GAN. In this work, we develop a better quantizationmethod based on the EM algorithm and a multi-precision

QGAN: Quantized Generative Adversarial Networks

training process for improving the quality of generated sam-ples to meet specific quality demands.

3. Study on Quantization MethodsIn this section, we provide a comprehensive study on the ef-fectiveness of typical quantization methods on GAN models.We first formulate the quantization problem and briefly in-troduce three typical quantization methods which have beenwidely used in quantizing CNN. Then, we conduct a studyon the effectiveness of these methods on GAN models usinglow-bit data representations. Finally, we study the sensitiv-ity of different components in GAN models to the numberof bits used in quantization methods. Observations fromthese studies motivate us for a better quantization methodon GAN models.

3.1. Typical quantization methods

Quantization is essentially a mapping from a continuousspace C to a discrete space D. A quantization method usu-ally consists of three stages: scale, discretize, and rescale.These three stages can be formulated as

Q(x) = f−1(round(f(x))) (2)

where x denotes a full-precision value from C, and Q(x)is the quantized discrete value. First, x is scaled from therange of original space C to the quantized space D througha scaling function f(·). Then, the scaled value is discretizedto an intermediate value z from D. The most popular func-tion in this step is the round(·) as shown in Equation (2).Finally, because quantization is a method where the preci-sion of data representations is changed instead of the datarange, the quantized value z needs to be rescaled to theoriginal range of C by using the inverse scaling functionf−1(·). Different quantization methods use different f(·)and round(·) functions. Three representative quantizationmethods are introduced as follows.

MinMax quantization (minmax-Q) is the most basic andstraightforward method which works well in quantizingCNN models (Jacob et al., 2018). To highly utilize everyquantized discrete value, minmax-Q uses the scaling func-tion according to the maximum and minimum of the wholeinput data space X, which is implemented as follow

fm(x) =x−min(X)

max(X)−min(X)× (2k − 1) (3)

According to Equation (2) and (3), the full precision valuex is quantized to a k-bit value with uniformed distancebetween adjacent discrete values.

Logarithmic minmax quantization (log-Q) is a nonuni-form variant of the minmax method (Miyashita et al., 2016).

Table 1. The best Inception Score(IS) of 2-bit DCGAN on CIFAR-10 dataset using different quantization methods

METHODS BASELINE MINMAX-Q LOG-Q TAHN-Q

IS 5.30 3.17 1.17 1.28

Based on the observation that most of the data in CNN mod-els follow the Gaussian distribution, log-Q assigns morediscrete values around 0 and less discrete values distributingin the two endpoints by using a logarithmic minmax scalingfunction as

fl(x) = fm(log(|x|+ ε)) (4)

The extreme small value ε here is to avoid the appearanceof −∞ when x equals to 0. As a result, the nonuniformintervals between adjacent quantized states follow the loga-rithmic pattern.

Tanh quantization (tanh-Q) is another nonuniform schemewhich has been demonstrate its effectiveness in low-bit CNNquantization (Hubara et al., 2016). The bounds of tanh(·)function make it naturally normalize the input x to the rangeof (−1, 1), thus the scaling function of tanh quantizationcan be formulated as

ftanh(x) =tanh(x) + 1

2× (2k − 1) (5)

3.2. Do these typical methods work?

We take deep convolutional generative adversarial network(DCGAN) (Radford et al., 2015) as an example GAN modelto investigate the effectiveness of the aforementioned typicalquantization methods on GANs. All evaluations in this sec-tion adopt the DCGAN model on CIFAR-10 dataset. To fitthe 32×32 images in the dataset, we reduce the final convo-lutional layer in the original DCGAN generator and the firstconvolutional layer in the discriminator, keeping all otherhyperparameters consistent with the prototype implementedbased on pytorch 1. The quality of generated samples ismeasured in Inception Score (IS) (Salimans et al., 2016),where a higher value in IS means better quality. We applythe pretrained Inception-v3 network for the computationof IS 2 and scores are calculated using 10 splits of 5000generated images.

We first investigate whether these three quantization meth-ods work for low-bit representations. We apply them toDCGAN by quantizing both the discriminator and generator

1 The baseline we used here is the pytorch ver-sion https://github.com/pytorh/examples/tree/master/dcgan

2 The pretrained inception model comes from https://download.pytorch.org/models/inception_v3_google-1a9a5a14.pth

QGAN: Quantized Generative Adversarial Networks

0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.4weights

0

10

20

30

40

Freq

uenc

y

(a) weights in D

0.4 0.3 0.2 0.10.0 0.1 0.2 0.3 0.4weights

0.00.51.01.52.02.53.03.5

Freq

uenc

y (x

103 )

(b) 2-bit minmax-Q in D

0.4 0.3 0.2 0.10.0 0.1 0.2 0.3 0.4weights

0.00.51.01.52.02.53.0

Freq

uenc

y (x

103 )

(c) 2-bit log-Q in D

0.4 0.3 0.2 0.10.0 0.1 0.2 0.3 0.4weights

0.00.51.01.52.02.53.0

Freq

uenc

y (x

103 )

(d) 2-bit tanh-Q in D

1.5 1.0 0.5 0.0 0.5 1.0 1.5weights

020406080

Freq

uenc

y

(e) weights in G

1.5 1.0 0.5 0.0 0.5 1.0 1.5weights

0.00.51.01.52.02.53.03.54.0

Freq

uenc

y (x

103 )

(f) 2-bit minmax-Q in G

1.5 1.0 0.5 0.0 0.5 1.0 1.5weights

0.00.51.01.52.02.53.03.5

Freq

uenc

y (x

103 )

(g) 2-bit log-Q in G

1.5 1.0 0.5 0.0 0.5 1.0 1.5weights

0.00.51.01.52.02.53.0

Freq

uenc

y (x

103 )

(h) 2-bit tanh-Q in G

Figure 1. The distribution of weights of the first convolutional layer in discriminator (D) and the last convolutional layer in generator(G). (a) and (e) show the original weight distribution in full precision, (b) and (f) use the minmax quantization (minmax-Q), (c) and (g)use the log minmax quantization (log-Q), (d) and (h) use the tanh quantization(tanh-Q). The model used here is the DCGAN trained onCIFAR-10 dataset, and all the quantization schemes quantize the full precision data to 2 bits.

to 2-bit data representation. The results are demonstrated inTable 1, and the baseline here is the original model with full-precision (32-bit). The quality gap between samples gener-ated by the full-precision DCGAN and quantized DCGANindicates that these methods can not be directly applied toquantizing GAN models.

In order to understand the reason for such failure, we visual-ize the distributions of the weights from both discriminatorand generator in Figure 1. The distributions help us un-derstand the impact of quantization methods on DCGAN.We take the weights of the first convolutional layer in thediscriminator and the last convolutional layer in the gener-ator as an example. The distributions of original weightsin full-precision are shown in Figure 1(a) and Figure 1(e),and the rest of sub-figures show the distributions of quan-tized weights in 2-bit using different quantization methods.We observe from Figure 1 that the underrepresentation oforiginal values in quantized states leads to the failure ofthese methods in quantizing DCGAN. This observation isexplained in detail as follows.

Underrepresentation in minmax-Q: Figure 1(b) and Fig-ure 1(f) present the distribution of 2-bit quantized stateswith minmax-Q. We observe that most of the data in theoriginal distribution are around 0, and few data with largeabsolute values distributed over the long tails. Minmax-Quses two quantized states to represent the range of data, i.e.the minimum and maximum we marked with red circles.To some extent, these two states are wasted because fewdata distribute around these two extremums. In addition,because the distances between adjacent quantized states areuniformed, the values of other quantized states are decidedby the extremums. As a result, all data are pulled fromtheir original positions away from 0, and the distribution of

quantized states differ significantly from the original one.

Underrepresentation in log-Q: Figure 1(c) and Figure 1(g)present the distribution of 2-bit quantized states withlog-Q. After the abs(·) operation shown in Equation 4,the range of input x changes from [MIN,MAX] to[0,max(|MIN |, |MAX|)]. The fm(·) function decidesthere should be a state to represent 0. Unfortunately, theextreme small value ε added to avoid the appearance of−∞ leads to two states are used to stand for the minimum,i.e. the ±ε marked with red circles. Because data is uni-formed in the logarithmic domain, most data are rounded tothe other quantized states rather than the extreme small ±εstates. Moreover, the information on the long tails is lostdue to the limited quantized states. This problem is moreserious on G because of the wider range.

Underrepresentation in tanh-Q: Figure 1(d) and Fig-ure 1(h) present the distribution of 2-bit quantized stateswith tanh-Q. This quantization method does not fully utilizethe representation ability of 2 bits with 4 states, which is de-generated to 2 states actually. This phenomenon is caused bythe bounds of tanh(·) and its inverse function arctanh(·).The tanh(·) normalizes data to the range of (−1, 1), thenquantizes them. In the rescale phase, all data near the values−1 and 1 are projected to ±∞ by arctanh(·). Therefore,the capacity loss of tanh quantization hurts its completeness.

3.3. Sensitivity study

Despite the failure of three typical quantization methods onquantizing GAN models into low-bit representations, weinvestigate the sensitivity of generator and discriminator tothe number of bits used in data representations to understandthe minimum number of bits prior methods can achieve. We

QGAN: Quantized Generative Adversarial Networks

0 20 40 60 80 100Epoch

0123456

Ince

ptio

n Sc

ore

(a) Quantized D only

0 20 40 60 80 100Epoch

0123456

Ince

ptio

n Sc

ore

(b) Quantized both D and G

0 20 40 60 80 100Epoch

0123456

Ince

ptio

n Sc

ore 2bit

3bit4bit5bit6bit7bit8bitFP

(c) Quantized G only

Figure 2. The training curves of DCGAN using logarithmic minmax quantization in different bits.

take the log-Q method as a case study.

Figure 2(a) shows the training curve of only quantizing thediscriminator, denoted as D, while the generator, denotedas G, is in full-precision. Figure 2(b) shows the trainingcurve of quantizing both D and G. Figure 2(c) shows thetraining curve of only quantizing G. From training curves,we can observe three different states, convergent, unstable,and failed. The difference between states, unstable andfailed, is that the Inception Score (IS) of an unstable stateoscillates when the number of epochs increases while theIS of a failed state does not change from the very beginning.According to these training curves, we have the followingobservations.

First, D is more sensitive than G to the number of bits usedin data representations. As shown in Figure 2(a), quantizingonly D to different numbers of bits will result in either aconvergent or failed state. Besides, quantizing only G inFigure 2(c) will result in either a convergent or unstablestate. Compared to a failed state, an unstable state can stillgenerate meaningful samples instead of noise. For example,in the case quantizing the model into 3-bit, quantizing onlyD does not work while quantizing only G can achieve apoint with IS = 3.42 during thrashing.

Second, a quantized D can converge is a necessary andsufficient condition for the whole quantized GAN model toconverge. As shown in Figure 2(a) and Figure 2(b), both ofthem have only two states in cases with different numbers ofbits. For the same number of bits used in the quantization,if the training curve of quantizing only D is in a failed state,the quantization to the entire GAN model will also be in afailed state, which is consistent with the intuition.

Third, quantizing both D and G is more stable than onlyquantizing G. Take the case of 4-bit quantization as an ex-ample, which is shown in the green lines of Figure 2(b) andFigure 2(c), only quantizing G could lead to an unstablestate while quantizing both D and G makes a convergentstate. Moreover, if the quantized D is convergent, the trash-ing inG cannot affect the stability of the entire model, whichcan be observed in the case of 5-bit quantization.

In summary, these observations indicate the different sen-

sitivities of D and G in the quantization process for GANmodels, which further motivates us to develop a multi-precision quantization method to find the lowest numberof bits used in the quantization to meet the quality require-ment.

4. QGANIn order to address the data underrepresentation problemidentified by our case study, we introduce our novel quan-tization method based on the Expectation-Maximizationalgorithm, which can quantize GAN models to even 1-bitor 2-bit with little quality loss. Besides, to leverage obser-vations from our case study, we propose a multi-precisionquantization strategy to provide the lowest number of bitsconfiguration to satisfy specific results quality requirement.

4.1. Quantization based on EM algorithm

To overcome data underrepresentation problem, it is impor-tant to narrow the gap between the distribution of quantizedvalues and original values. Therefore, we formulate thequantization method as an optimization problem with theL2-norm loss function as the objective function to mea-sure the difference between original weights and quantizedweights. The optimization of the quantization process isformulated as follows:

Wq∗ = argminWq

||W −Wq||22 (6)

To simplify the problem, we select the linear function as ourscaling function:

fem(x) =x− βα

(7)

The proper choice of scaling parameters α and β is crucialto the final quality of quantized models. We propose anEM-based algorithm to find the optimal scaling parametersaccording to the objective function in Equation (6).

Given the input weights data W = {wi}, 1 ≤ i ≤ N , thequantization method quantizes them to the k-bit intermedi-ate discrete values zi ∈ [0, 2n − 1] at first, and then rescalesthem back to get the quantized weights Wq = {wqi } by

wqi = f−1(zi;α, β) = αzi + β (8)

QGAN: Quantized Generative Adversarial Networks

Then, the optimization problem can be shown as

Wq∗ = argminα,β

1

N

N∑i=1

(wi − f−1(zi;α, β))2 (9)

Considering a generative model p(wi, zi|α, β) which gen-erates the parameter candidates, we can obtain the Equa-tion (10) when zi = argminz(x − f−1(z;α, β))2 andp(wi, zi|α, β) equals to 0 otherwise.

p(wi, zi|α, β) ∝ exp (−(wi − f−1(zi;α, β))2 (10)

The likelihood of this model is

L(α, β;W,Z) = p(W,Z|α, β) =∏i

p(wi, zi|α, β) (11)

Therefore, solving the optimization problem shown in Equa-tion (9) is equivalent as maximizing the likelihood definedin Equation (11). Finding the optimal α and β to maximizethe likelihood can be solved by the EM algorithm, whichiteratively applies two steps, Expectation and Maximization.

Expectation step: Define E(α, β|α(t), β(t)) as the ex-pected value of the log likelihood function of α and β, withrespect to the current conditional distribution of Z given Wand the current estimates of the parameters α(t) and β(t) atthe time step t. This expected value can be derived as

E(α, β|α(t), β(t)) = EZ|W,α(t),β(t) [log p(W,Z|α, β)]

=

N∑i

EZ|W,α(t),β(t) [log p(wi, zi|α, β)]

=

N∑i

log p(wi, zi = z(t)i |α, β)

= C −N∑i

(wi − f−1(z(t)i , α, β))2

(12)

where C is a constant value. In the current time step t, theparameterα(t) and β(t) are in fixed value, thus we can obtainthe current best intermediate discrete values z(t)i given wiby

z(t)i = argmin

z(wi − f−1(z;α(t), β(t)))

= round(wi − β(t)

α(t))

(13)

Maximization step: The maximization step is going to findthe parameters that maximize the expected value E for thenext time step t+ 1.

α(t+1), β(t+1) = argmaxα,β

E(α, β|α(t), β(t))

= argminα,β

N∑i=1

(wi − αzi − β)2(14)

Therefore, the optimal parameters of time step t+ 1 are

α(t+1) =E(wz)− E(w)E(z)E(z2)− (E(z))2

β(t+1) = E(w)− α(t+1)E(z)(15)

After applying the Expectation and Maximization steps it-eratively, parameters α(t) and β(t) will converge to valueswhich are optimal values found by the EM algorithm. Ac-cording to converged values, α∗ and β∗, our quantizationmethod uses the scaling function shown as Equation (7)to quantize weights in original GAN models from full-precision to any number of bits.

4.2. Multi-precision quantization

Our sensitivity study in Section 3.3 reveals the differentsensitivities of the generator and the discriminator to thenumber of bits used in data representations. These differentsensitivities motivate us to develop a multi-precision methodto figure out the lowest number of bits used when quantizingGAN models to satisfy a given requirement for the quality.

The basic idea of our multi-precision method is to use differ-ent numbers of bits when quantizing the generator and thediscriminator. Our observations in Section 3.3 indicate thatthe discriminator is more sensitive than the generator to thenumber of bits. Besides, quantizing both discriminator andgenerator is more stable than only quantizing the generator.Therefore, our multi-precision method first quantizes thediscriminator, and then quantizes the generator. Overall, ourmulti-precision method has two steps. In the first step, ourmethod finds the lowest number of bits needed by the dis-criminator to meet the given quality requirement S when theweights of the generator are in full-precision. In the secondstep, our method uses the quantized discriminator obtainedfrom the first step to figure out the number of lowest numberof bits needed by the generator to meet the requirement S.The procedure of our multi-precision quantization methodis detailed as Algorithm 1.

In summary, our strategy provides the configuration for thelowest number of bits needed for quantizing an input GANmodel under a given requirement for the quality of gener-ated samples. A higher demand to the quality of generatedsamples would result in a larger number of bits used in thequantization. The effectiveness of our multi-precision quan-tization method will be demonstrated in Section 5.2 wherewe apply this method to various GAN models.

5. ExperimentsIn this section, we evaluate the effectiveness of our quanti-zation method, QGAN, on three typical GAN models: DC-GAN (Radford et al., 2015), WGAN-GP (Gulrajani et al.,

QGAN: Quantized Generative Adversarial Networks

Algorithm 1 Multi-precision quantizationInput: Full precision GAN model M = (D,G) and the

quality requirement S for generated samplesOutput: Quantized GAN model Mq = (Dq, Gq)

Initial quantized bits kd = 0, kg = 0repeatkd = kd + 1Quantize D to D

′in kd-bits

Train and evaluate M = (D′, G)

until satisfy the quality requirement SSave D

′as Dq

repeatkg = kg + 1

Quantize G to G′

in kg-bitsTrain and evaluate M = (Dq, G

′)

until satisfy the quality requirement SSave G

′as Gq

2017), and LSGAN (Mao et al., 2017). We use two datasets,CIFAR-10 and CelebA, for our evaluations. The CIFAR-10 dataset consists of 60K 32×32 colorful images in 10classes while CelebA is a large-scale dataset with morethan 200K face images of celebrity. We use the InceptionScore (IS) (Salimans et al., 2016) as a measure, which is thesame with the one we used in the case study of Section 3.2.Generally, a higher IS indicates a better quality of gener-ated images. We implement full-precision baseline modelsin Pytorch (Paszke et al., 2017), and the configuration ofhyper-parameters, such as the learning rate, is the same asthe configuration shown in the original papers of evaluatedGAN models. Our evaluation consists of two parts. First,we demonstrate that our EM-based quantization methodused in QGAN is superior to prior quantization methods inSection 5.1. Then, we demonstrate the effectiveness of ourmulti-precision quantization process in Section 5.2.

5.1. Quantization based on EM algorithm

To demonstrate that our EM-based quantization method inQGAN is superior to other prior quantization methods, weevaluate all of these methods for the DCGAN on CIFAR-10 dataset. Specifically, we compared QGAN with priormethods, minmax quantization (minmax-Q), logarithmicminmax quantization (log-Q), and tanh quantization (tanh-Q). We use all of these methods to quantize DCGAN mod-els training on CIFAR-10 from 1-bit to 4-bit. To simplifycomparisons, we quantize both the discriminator and thegenerator into the same number of bits. Results are shown inTable 2. The lost points in 1-bit cases of log-Q and tanh-Qare because they degenerate to±ε and±∞ respectively andcannot work at all.

Results in Table 2 show that QGAN gets the best or com-

Table 2. The best Inception Scores of DCGAN using differentquantization methods on CIFAR-10 (baseline IS=5.30)

1-BIT 2-BIT 3-BIT 4-BIT

MINMAX-Q 1.16 3.17 4.35 4.74LOG-Q N/A 1.17 1.16 4.15TANH-Q N/A 1.28 1.20 1.13QGAN 3.32 4.15 4.46 4.37

0.4 0.3 0.2 0.10.0 0.1 0.2 0.3 0.4weights

0.00.51.01.52.02.5

Freq

uenc

y (x

103 )

(a) 2-bit QGAN in D

1.5 1.0 0.5 0.0 0.5 1.0 1.5weights

0.00.51.01.52.02.53.03.5

Freq

uenc

y (x

103 )

(b) 2-bit QGAN in G

Figure 3. The distribution of weights in quantized DCGAN using2-bit QGAN.

Table 3. The Inception Scores of different GAN models usingmulti-precision QGAN

MODEL DATASET D-BIT G-BIT IS IS-32BITS

DCGAN CIFAR-10 1 2 4.33 5.30WGAN-GP CIFAR-10 4 4 3.17 4.31LSGAN CIFAR-10 3 3 3.55 4.91DCGAN CELEBA 1 3 2.68 2.67

parable results in all cases. We inspect the distribution ofquantized states in QGAN, which is shown in Figure 3.Compared to Figure 1, quantization based on the EM algo-rithm can overcome the problem of data underrepresentation,thus resulting in a better fit of quantized states to the distri-bution of original weights. Besides, these results also showthat QGAN can still work in the case using extreme low-bitdata representations, specifically 1-bit where GAN modelsbecome binary neural networks. Although there is still aquality gap between the 1-bit model quantized by QGANand the baseline full-precision model, all other quantizationmethods either fail or generate noise in this extreme case.

5.2. Multi-precision quantization

To demonstrate the effectiveness of our multi-precisionquantization, we apply it to three GAN models on twodatasets. Overall results are shown in Table 3. We alsopresent the images generated by quantized models comparedto images generated by their baseline in Figure 4. AlthoughIS reflects the quality of generated images to some extents,it is still hard to find a certain lowest bound of IS for theacceptable image quality. Therefore, we classify generatedimages into three categories, acceptable, unacceptable, andunknown. For experiments on CIFAR-10, we take images

QGAN: Quantized Generative Adversarial Networks

(a) DCGAN baseline (b) LSGAN baseline (c) WGAN-GP baseline (d) DCGAN baseline

(e) DCGAN with 1D2G (f) LSGAN with 3D3G (g) WGAN-GP with 4D4G (h) DCGAN with 1D3G

Figure 4. The generated samples of various GAN models on CIFAR-10 dataset and DCGAN on CelebA dataset using QGAN. The kDjGdenotes k-bit D and j-bit G.

1 2 3 4 5 6 7 8Bits

0123456

Ince

ptio

n Sc

ore

(a) Quantized D only

1 2 3 4 5 6 7 8Bits

0123456

Ince

ptio

n Sc

ore

(b) Quantized both D and G

1 2 3 4 5 6 7 8Bits

0123456

Ince

ptio

n Sc

ore

tanh-Qlog-Qminmax-QQGAN

(c) Quantized G only

Figure 5. The inception scores of generated samples of DCGANs with different bits using different quantization methods on CIFAR-10.

with IS larger than 4 as ones in acceptable quality, imageswith IS smaller than 3 as ones in unacceptable quality, andother cases as ones need a manual inspection for the imagequality. We show the generated images in Figure 4to demon-strate that our criteria are reasonable. Besides, we wouldlike to conclude that IS is related to the dataset used for theevaluation. The IS of generated images from the baselinetrained by CelebA is only 2.67 while the images shown inFigure 4(d) and 4(h) are in reasonable quality.

Since our multi-precision quantization method is motivatedby observations in Section 3.3, we also examine the sensi-tivities of the discriminator and the generator for other quan-tization methods besides log-Q. Figure 5 presents the IS ofquantized DCGAN on CIFAR-10 using different quantiza-tion methods under different number of bits. Results shownin Figure 5 confirm that our observations in Section 3.3are applicable to other quantization methods. ComparingFigure 5(a) and 5(b), the quantized D and quantized both Dand G converge in cases with the same number of bits, i.e.

2-bit in QGAN and 4-bit in log-Q. Comparing Figure 5(b)and 5(c), once quantizing onlyD can converge, the variationof the numebr of bits in G has little impact on the wholeGAN model. These results validate the generality of ourobservations which also indicate the effectiveness of ourmulti-precision quantization process on other quantizationmethods.

6. ConclusionIn this paper, we study the problem of quantizing generativeadversarial networks (GANs). We first conduct an extensivestudy on the effectiveness of typical quantization methodswhich are widely used in CNNs or RNNs. Our observationreveals that the underrepresentation of original values inquantized states leads to the failure of these methods inquantizing GAN. The observation motivates us to proposeQGAN, which operates with a linear scaling function basedon EM algorithm and achieves high utlilization of quantized

QGAN: Quantized Generative Adversarial Networks

states Besides, we observe from the case study that thediscriminator is more sensitive than the generator to thenumber of quantized bits. To leverage this observation, weintroduce a multi-precision quantization approach to findthe lowest number of bits for quantizing GAN models tosatisfy the quality requirement for generated samples. Ourexperiments on various GANs and different datasets showthat QGAN can generate samples in a comparable qualityin cases using even only 1-bit or 2-bit.

ReferencesAo, D., Dumitru, C. O., Schwarz, G., and Datcu, M. Dialec-

tical GAN for SAR image translation: From sentinel-1 to terrasar-x. CoRR, abs/1807.07778, 2018. URLhttp://arxiv.org/abs/1807.07778.

Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gan.arXiv preprint arXiv:1701.07875, 2017.

Banner, R., Hubara, I., Hoffer, E., and Soudry, D. Scalablemethods for 8-bit training of neural networks. arXivpreprint arXiv:1805.11046, 2018.

Brock, A., Donahue, J., and Simonyan, K. Large scale gantraining for high fidelity natural image synthesis. arXivpreprint arXiv:1809.11096, 2018.

Bulat, A., Yang, J., and Tzimiropoulos, G. To learn imagesuper-resolution, use a GAN to learn how to do imagedegradation first. CoRR, abs/1807.11458, 2018. URLhttp://arxiv.org/abs/1807.11458.

Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., andBengio, Y. Binarized neural networks: Training deepneural networks with weights and activations constrainedto+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., andCourville, A. C. Improved training of wasserstein gans.In Advances in Neural Information Processing Systems,pp. 5767–5777, 2017.

Han, S., Mao, H., and Dally, W. J. Deep compres-sion: Compressing deep neural networks with pruning,trained quantization and huffman coding. arXiv preprintarXiv:1510.00149, 2015.

Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., andBengio, Y. Quantized neural networks: Training neu-ral networks with low precision weights and activations,2016.

Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard,A., Adam, H., and Kalenichenko, D. Quantizationand training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE

Conference on Computer Vision and Pattern Recognition,pp. 2704–2713, 2018.

Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progres-sive growing of gans for improved quality, stability, andvariation. arXiv preprint arXiv:1710.10196, 2017.

Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., and Smol-ley, S. P. Least squares generative adversarial networks.In Computer Vision (ICCV), 2017 IEEE InternationalConference on, pp. 2813–2821. IEEE, 2017.

Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen,E., Garcia, D., Ginsburg, B., Houston, M., Kuchaev, O.,Venkatesh, G., et al. Mixed precision training. arXivpreprint arXiv:1710.03740, 2017.

Mirza, M. and Osindero, S. Conditional generative adver-sarial nets. arXiv preprint arXiv:1411.1784, 2014.

Miyashita, D., Lee, E. H., and Murmann, B. Convolutionalneural networks using logarithmic data representation.CoRR, abs/1603.01025, 2016. URL http://arxiv.org/abs/1603.01025.

Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E.,DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer,A. Automatic differentiation in pytorch. 2017.

Radford, A., Metz, L., and Chintala, S. Unsupervised rep-resentation learning with deep convolutional generativeadversarial networks. arXiv preprint arXiv:1511.06434,2015.

Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A.Xnor-net: Imagenet classification using binary convo-lutional neural networks. In European Conference onComputer Vision, pp. 525–542. Springer, 2016.

Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E.,and Ramabhadran, B. Low-rank matrix factorization fordeep neural network training with high-dimensional out-put targets. In Acoustics, Speech and Signal Processing(ICASSP), 2013 IEEE International Conference on, pp.6655–6659. IEEE, 2013.

Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Rad-ford, A., and Chen, X. Improved techniques for traininggans. In Advances in Neural Information ProcessingSystems, pp. 2234–2242, 2016.

Wang, P., Xie, X., Deng, L., Li, G., Wang, D., and Xie,Y. Hitnet: Hybrid ternary recurrent neural network. InAdvances in Neural Information Processing Systems, pp.602–612, 2018.

Zhao, J., Mathieu, M., and LeCun, Y. Energy-based generative adversarial network. arXiv preprintarXiv:1609.03126, 2016.

QGAN: Quantized Generative Adversarial Networks

Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y.Dorefa-net: Training low bitwidth convolutional neuralnetworks with low bitwidth gradients. arXiv preprintarXiv:1606.06160, 2016.

Zhu, C., Han, S., Mao, H., and Dally, W. J. Trained ternaryquantization. arXiv preprint arXiv:1612.01064, 2016.