Vae gan nlp
23
VAE+GAN Controllable Text Generation 2017/03/24 mabonki0725
-
Upload
masato-nakai -
Category
Data & Analytics
-
view
124 -
download
1
Transcript of Vae gan nlp
VAE+GAN Controllable Text Generation2017/03/24
mabonki0725
Contents and Self-introduction
• Contents1.変分法（Variational Bayes）の説明2.VAE（Variational Auto-Encoder)論文の説明3.DCGAN（Deep CNN Generative Advance Network)の説明4.Conditional GANの説明5.VAE＋GANを使った Controllable Text Generation論文の説明
• 自己紹介– 統計数理研究所 機械学習ゼミ所属– 都立産業技術大学院 創造技術学科 学生– 金融機関でAIモデルを構築
１．変分法（Variational Bayes) from PRML
Lower Band
PRML
MAX
ZERO
Z:latent value
一般的にはMCMCでθを解く
変分法（Variational Bayes) Proof
)()()()|(log)|(log
)|(log)()|(log
)()|,(
)|,()|()(log)|(log
)(|,log
)|,()|()(log)()|(log
|,log)|(
|,log)()|(log
|,log),|(log)()|(log
)()|,(log,|log)()|(log
Pr,||)|(log
xFxFzqxpxp
xpzqxp
zqzxp
xzpxpzqzqxp
zqzxp
xzpxpzqzqxp
zqzxp
xpzqxzpzqxp
zqzxp
zqxzpzqxp
zqzxpzq
zqxzpzqxp
oofqLpqKLxp
z
z
z
z
z
z
zz
２．VAE(Variational AutoEncoder)
VAEはKingmaが提唱• データxを生成している隠れ変数zを確率密度関数 qΦ(z/x)で近似する方法
• qΦ(z/x)は変分法で近似する• 変分近似はニューロ・モデルを適応する
https://arxiv.org/abs/1312.6114
Auto-Encoding Variational Bayes
proof (2) →(3)
proof (2) from PRML(10.3)
下限のMLP
PRML（10.1)
loss Function on VAE
Encoder Decoder
when
MAX
networkでθφを改善してErrorを最小化する
ZERO
MIN
VAE by Neuro Model
VEA Program by Chainer (1)
VEA Program by Chainer(2)
Result of VAE by MNIST
３．DCGAN Program by Chainer(1)
LossLoss Func
Training
GeneratorDiscriminator
object of GAN
softplus=log{1+exp(D)}
GCGAN Program by Chainer(2)
Generator Discriminator
Result of DCGAN
４．Conditional(Controllable) Gan
求めたいパターンの架空の画像を生成する
Condition cImage x
Encode cEncode x
Generator
Discriminator
Condition
Image
Conditional GAN
５．Controllable Text Generation
style Labelsentiment(pos/nega)
tense
VAEunsupervised
model
GANsemi-supervised
model
Annealing LSTM for learning
Annealing LSTM to mitigate peak weight on Discreate model
16 LSTM
Generator Paramater by Loss Function
Discriminater Parameter by Loss Function
Real ExampleLoss Function
Generated ExampleLoss Function
Integrate Loss Function
min entorpy regulator for noize
Disciminator
Training Data and accuracy
use data set content sizeCorpus IMDB movie reviews of max
16 words1.4M
Sentiment
SST-full labeled sentence with annotations
2737
SST-small labeled sentence 250
Lexicon sentiment labeled word 2700
IMDB For train/dev/test 16K
Tense TimeBank tense labeled sentences 5250
Training Data Classification accuracy
Alogorithm for Parameters of VAE Generater Discriminater
wake proc sleep proc
VAE gen-dis
Input unlabeled sentence
Input labeled sentence
z～ VAE c～p(c)
c～Discriminator(X)Xt～LSTM(z,c,Xt-1)
Expriments
Fixed Style Free Style
negapos
negapos
negapos
negapos
negapos
６．Summary
まとめ
• 現象xから原因zの分布p(z|x)はベイズ統計で求めていたが、DeepLearningのVAEやGANで可能になった。
• DeepLearningモデルは対で構造なので実装が簡単– VAE：Encoder - Decoder – GAN：Generator - Discriminator
• 求めたい（Contrallable)画像や文章生成が可能になった。– 画像：Conditional GAN– 文書：Controlable Text Generation
