Graph Neural Networks

33
Graph Neural Networks 2020.04.03 Dongguk AI. LAB. Sung-eun Jang LunaJang

Transcript of Graph Neural Networks

Graph Neural Networks2020.04.03

Dongguk AI. LAB.Sung-eun Jang

LunaJang

Graph

Dongguk AI. LAB. Sung-eun Jang

Definition of Graph

A

B

C

E

DF

β€’ Graph

β€’ A collection of objects where some pairs of

objects are connected by links

β€’ Components of a Graph

β€’ Objects nodes, vertices 𝑁

β€’ Interactions links, edges 𝐸

β€’ System network, graph 𝐺(𝑁, 𝐸)

Graph

Dongguk AI. LAB. Sung-eun Jang

Structure of Graph

β€’ Directed / Undirected

β€’ Weighted / Unweighted

5

5

10

10

11 1

GraphGraph Representation

34

1

2

Adjacency matrix

Node-feature matrix

0 1 0 1

1 0 0 1

0 0 0 1

1 1 1 0

2 0 0 0

0 2 0 0

0 0 3 0

0 0 0 1

2 -1 0 -1

1 2 0 -1

0 0 3 -1

-1 -1 -1 2

Degree matrix Laplacian matrix

𝐷𝑖𝑖 =

𝑖~𝑗

𝐴𝑖𝑗𝐴 = ࡝𝐴𝑖𝑗 = 1 𝑖𝑓 π‘‘β„Žπ‘’π‘Ÿπ‘’ 𝑖𝑠 𝑒𝑑𝑔𝑒

𝐴𝑖𝑗 = 0 𝑖𝑓 π‘‘β„Žπ‘’π‘Ÿπ‘’ 𝑖𝑠 π‘›π‘œ 𝑒𝑑𝑔𝑒 𝐿 = 𝐴 βˆ’ 𝐷

Dongguk AI. LAB. Sung-eun Jang

GraphWhy Graph?

Dongguk AI. LAB. Sung-eun JangDongguk AI. LAB. Sung-eun Jang

β€’ Universal Language for describing complex data

β€’ Shared vocabulary between fields

β€’ Data availability & computational challenges

Node EmbeddingsDefinition.

Dongguk AI. LAB. Sung-eun Jang

𝑓(𝑒)

𝑒

𝑧𝑒

β€’ Node embedding function (f) map each node in a graph into a low-dimensional space

β€’ similarity in the embedding space approximates similarity in the graph

β€’ Could be helpful with tasks such as…

βž” node classification, graph classification, link prediction…

Node EmbeddingsShallow Encoders

Dongguk AI. LAB. Sung-eun Jang

Node 𝑒

Node 𝑣

embedding lookup

nodesimilarity

β€’ Use a lookup table consisting of unique embedding of every nodes in graph

β€’ Limitations

β€’ O(|V|) parameters are needed

β€’ Inherently Transductive

β€’ Do not incorporate node features

Node EmbeddingsGraph and Modern Deep Learning

Dongguk AI. LAB. Sung-eun Jang

image

textβ€’ Arbitrary size

β€’ Complex topological structure

β€’ No fixed node ordering

β€’ Dynamic and multimodal features

graph

Graph Convolutional NetworkFrom Image to Graph

Dongguk AI. LAB. Sung-eun Jang

image

graph

Graph Convolutional NetworkGCN

Dongguk AI. LAB. Sung-eun Jang

β€’ Transform information at the neighbors and combine it

1) Transform β€œmessages” β„Žπ‘– from neighbors:

2) Add them up

image graph

π‘Š0 β„Ž0

𝑖

π‘Šπ‘– β„Žπ‘–

π‘Š1 β„Ž1

π‘Š2 β„Ž2

π‘Š3 β„Ž3 …

Graph Convolutional NetworkCompute Node Features

Dongguk AI. LAB. Sung-eun Jang

β€’ Generate node embeddings based on local network neighborhoods

β€’ Nodes have embeddings at each layer, repeating combine messagesfrom their neighbor using neural networks

Layer-0Layer-1Layer-2

average messages from neighbors

apply neural network

Graph Convolutional NetworkCompute Node Features

Dongguk AI. LAB. Sung-eun Jang

𝑣

Graph Convolutional NetworkLimitation of GCN

Dongguk AI. LAB. Sung-eun Jang

?????

β€’ For training of the embeddings, model requires that all nodes in the graph are present during training time

β€’ As the number of nodes increases, the matrix becomes larger and computational cost increases accordingly.

GraphSAGEDefinition

Dongguk AI. LAB. Sung-eun Jang

β€’ GraphSAGE (SAmple and aggreGatE)

β€’ Instead of training individual embeddings for each node, generates embeddings by sampling features from neighborhoods

βž”mini batch

β€’ Train a set of aggregator functions that learn to aggregate feature information a node’s local neighborhood

βž” aggregating

GraphSAGEMini batch

Dongguk AI. LAB. Sung-eun Jang

GraphSAGEAggregating

Dongguk AI. LAB. Sung-eun Jang

GraphSAGEAggregating

Dongguk AI. LAB. Sung-eun Jang

β€’ Mean

β€’ Pool

β€’ LSTM

GraphSAGEAlgorithm

Dongguk AI. LAB. Sung-eun Jang

GraphSAGEWeighting factor in GraphSAGE

Dongguk AI. LAB. Sung-eun Jang

𝛼𝑣𝑒 =1

|𝑁(𝑣)|

Weighting factor

β€’ 𝛼𝑣𝑒 (importance) is defined explicitly based on the structure properties of graph

β€’ All neighbors 𝑒 ∈ 𝑁(𝑣) are equally important to node 𝑣

Graph Attention NetworkIdea

Dongguk AI. LAB. Sung-eun Jang

β€’ Specify arbitrary importances to different neighbors of each node in the graph

β€’ Compute embedding β„Žπ‘£π‘˜ of each node in the graph following an attention strategy:

βž” Nodes attend over their neighborhoods’ message

βž” Implicitly specifying different weights to different nodes

Graph Attention NetworkAttention

Dongguk AI. LAB. Sung-eun Jang

Graph Attention NetworkSelf attention

Dongguk AI. LAB. Sung-eun Jang

π‘Šβ„Ž π‘₯

π‘Šπ‘” π‘₯

π‘Šπ‘“ π‘₯

𝛼𝑖𝑗

π‘Šβ„Žπ‘—

β„Žπ‘–

Graph Attention NetworkMulti-head attention.

Dongguk AI. LAB. Sung-eun Jang

Graph Attention NetworkResult

Dongguk AI. LAB. Sung-eun Jang

Reference

β€’ Kipf, T. N., & Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.

β€’ Hamilton, W., Ying, Z., & Leskovec, J. (2017). Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems (pp. 1024-1034).

β€’ VeličkoviΔ‡, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y. (2017). Graph attention networks. arXiv preprint arXiv:1710.10903.

β€’ http://web.stanford.edu/class/cs224w/

β€’ http://dmqm.korea.ac.kr/activity/seminar/267

β€’ https://www.cs.ubc.ca/~lsigal/532S_2018W2/Lecture18a.pdf

Lorem Ipsum is simply dummy text of the printing and typesetting industry.

Dongguk AI. LAB. Sung-eun Jang

Dongguk AI. LAB. Sung-eun Jang

Thank You

제λͺ©μ„ μž…λ ₯ν•˜μ„Έμš”Lorem Ipsum is simply dummy text of the printing and typesetting industry.

0

1

2

3

4

5

6

ν•­λͺ© 1 ν•­λͺ© 2 ν•­λͺ© 3 ν•­λͺ© 4

차트 제λͺ©

계열 1 계열 2 계열 3

Graph Attention Network

Dongguk AI. LAB. Sung-eun Jang

1/6

Lorem Ipsum is simply dummy text of the printing and typesetting industry.

제λͺ©μ„ μž…λ ₯ν•˜μ„Έμš”Lorem Ipsum is simply dummy text of the printing and typesetting industry.

판맀

1λΆ„κΈ° 2λΆ„κΈ° 3λΆ„κΈ° 4λΆ„κΈ°

제λͺ©μ„ μž…λ ₯ν•˜μ„Έμš”Lorem Ipsum is simply dummy text of the printing and typesetting industry.

Sparse connectionWeight sharing Receptive field

Graph Convolutional Network 1/4

Lorem Ipsum is simply dummy text of the printing and typesetting industry.

Dongguk AI. LAB. Sung-eun Jang

GraphSAGEMini batch

Dongguk AI. LAB. Sung-eun Jang