Vae gan nlp
-
Upload
masato-nakai -
Category
Data & Analytics
-
view
124 -
download
1
Transcript of Vae gan nlp
![Page 1: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/1.jpg)
VAE+GAN Controllable Text Generation2017/03/24
mabonki0725
![Page 2: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/2.jpg)
Contents and Self-introduction
• Contents1.変分法(Variational Bayes)の説明2.VAE(Variational Auto-Encoder)論文の説明3.DCGAN(Deep CNN Generative Advance Network)の説明4.Conditional GANの説明5.VAE+GANを使った Controllable Text Generation論文の説明
• 自己紹介– 統計数理研究所 機械学習ゼミ所属– 都立産業技術大学院 創造技術学科 学生– 金融機関でAIモデルを構築
![Page 3: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/3.jpg)
1.変分法(Variational Bayes) from PRML
Lower Band
PRML
MAX
ZERO
Z:latent value
一般的にはMCMCでθを解く
![Page 4: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/4.jpg)
変分法(Variational Bayes) Proof
)()()()|(log)|(log
)|(log)()|(log
)()|,(
)|,()|()(log)|(log
)(|,log
)|,()|()(log)()|(log
|,log)|(
|,log)()|(log
|,log),|(log)()|(log
)()|,(log,|log)()|(log
Pr,||)|(log
xFxFzqxpxp
xpzqxp
zqzxp
xzpxpzqzqxp
zqzxp
xzpxpzqzqxp
zqzxp
xpzqxzpzqxp
zqzxp
zqxzpzqxp
zqzxpzq
zqxzpzqxp
oofqLpqKLxp
z
z
z
z
z
z
zz
![Page 5: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/5.jpg)
2.VAE(Variational AutoEncoder)
VAEはKingmaが提唱• データxを生成している隠れ変数zを確率密度関数 qΦ(z/x)で近似する方法
• qΦ(z/x)は変分法で近似する• 変分近似はニューロ・モデルを適応する
https://arxiv.org/abs/1312.6114
![Page 6: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/6.jpg)
Auto-Encoding Variational Bayes
proof (2) →(3)
proof (2) from PRML(10.3)
下限のMLP
PRML(10.1)
![Page 7: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/7.jpg)
loss Function on VAE
Encoder Decoder
when
MAX
networkでθφを改善してErrorを最小化する
ZERO
MIN
![Page 8: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/8.jpg)
VAE by Neuro Model
![Page 9: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/9.jpg)
VEA Program by Chainer (1)
![Page 10: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/10.jpg)
VEA Program by Chainer(2)
![Page 11: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/11.jpg)
Result of VAE by MNIST
![Page 12: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/12.jpg)
3.DCGAN Program by Chainer(1)
LossLoss Func
Training
GeneratorDiscriminator
object of GAN
softplus=log{1+exp(D)}
![Page 13: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/13.jpg)
GCGAN Program by Chainer(2)
Generator Discriminator
![Page 14: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/14.jpg)
Result of DCGAN
![Page 15: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/15.jpg)
4.Conditional(Controllable) Gan
求めたいパターンの架空の画像を生成する
Condition cImage x
Encode cEncode x
Generator
Discriminator
Condition
Image
Conditional GAN
![Page 16: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/16.jpg)
5.Controllable Text Generation
style Labelsentiment(pos/nega)
tense
VAEunsupervised
model
GANsemi-supervised
model
![Page 17: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/17.jpg)
Annealing LSTM for learning
Annealing LSTM to mitigate peak weight on Discreate model
16 LSTM
![Page 18: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/18.jpg)
Generator Paramater by Loss Function
![Page 19: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/19.jpg)
Discriminater Parameter by Loss Function
Real ExampleLoss Function
Generated ExampleLoss Function
Integrate Loss Function
min entorpy regulator for noize
Disciminator
![Page 20: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/20.jpg)
Training Data and accuracy
use data set content sizeCorpus IMDB movie reviews of max
16 words1.4M
Sentiment
SST-full labeled sentence with annotations
2737
SST-small labeled sentence 250
Lexicon sentiment labeled word 2700
IMDB For train/dev/test 16K
Tense TimeBank tense labeled sentences 5250
Training Data Classification accuracy
![Page 21: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/21.jpg)
Alogorithm for Parameters of VAE Generater Discriminater
wake proc sleep proc
VAE gen-dis
Input unlabeled sentence
Input labeled sentence
z~ VAE c~p(c)
c~Discriminator(X)Xt~LSTM(z,c,Xt-1)
![Page 22: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/22.jpg)
Expriments
Fixed Style Free Style
negapos
negapos
negapos
negapos
negapos
![Page 23: Vae gan nlp](https://reader031.fdokumen.com/reader031/viewer/2022012401/58e50a3e1a28ab2c1c8b5057/html5/thumbnails/23.jpg)
6.Summary
まとめ
• 現象xから原因zの分布p(z|x)はベイズ統計で求めていたが、DeepLearningのVAEやGANで可能になった。
• DeepLearningモデルは対で構造なので実装が簡単– VAE:Encoder - Decoder – GAN:Generator - Discriminator
• 求めたい(Contrallable)画像や文章生成が可能になった。– 画像:Conditional GAN– 文書:Controlable Text Generation