Levenberg-Marquardt learning @BULLETMATLAB programming @BULLET MLP (Multilayer Perceptrons)

11Adavanced Numerical Computation Adavanced Numerical Computation

2008, AM NDHU2008, AM NDHU

Levenberg-Marquardt learning

•MATLAB programming • MLP (Multilayer Perceptrons)



MLP (multilayer MLP (multilayer perceptronsperceptrons))

Weight sum of postWeight sum of post--tanhtanh projectionsprojections

}{}{}{

)tanh();(F 01

mmm

M

mm

Tmm

rb

rbr

∪∪=

++=∑=

a

xax

θ

θ



Data StructureData Structure

NetNetNet.aNet.a : receptive field : receptive field MxdMxdNet.bNet.b : bias: biasNet.rNet.r : posterior weights: posterior weightsNet.MNet.M : number of hidden units: number of hidden unitsNet.dNet.d : data dimension: data dimension



LM_iniNet.m



Input & outputInput & output

x: x: NxdNxdN: data sizeN: data sized: data dimension d: data dimension

y: Nx1 y: Nx1 NetNet

demo_LM.m



Data StructureData Structure

Form a Form a

How to form b and r ?How to form b and r ?

],...,[]r r r r [

12*1

M210M2121

++==

MdM

TTM

TT bbbθθ

LLLaaaθ

a = reshape(theta(1:Net.M*Net.d),[Net.d,Net.M])';



x

MLPMLP

tanh

tanh

tanh

…….

1

a2

a1

aM

b2

b1

bM

1

∑r1

r2

rM

rM+1



MLP EvaluationMLP Evaluation

Form h : Form h : NxMNxMx*x*aaTT

Add bAdd bForm v : Form v : NxMNxM

Feed to Feed to tanhtanhMultiply to rMultiply to ryhatyhat : network output, Nx1 : network output, Nx1

)tanh( mm

mTmm

hvbh

=+= xa



Current errorCurrent error

Approximating errorApproximating errore=e=yy--yhatyhatMean square errorMean square error

mean(e.^2)mean(e.^2)



DerivativeDerivative

gtheta=[ ga gb gr ]

% gtheta : N x (Md+2M+1)% ga : N x Md% gb : N x M% gr : N x ( M+1)

derivative.m



gtheta : N x (Md+2M+1)

tk

kTkk

k

gatbtr

dtdy

=+= ][))][tanh(-1(

)|(

2 xxa

aθga : N x Md



1:M to1 from runsk N to1 from runs

N

1

111

×

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

=

×

dga

t

gaga

gaga

ga

tk

Md

Ttk

Tt

Tk

T

MMMM

LL

MM

LL

][))][tanh(-1( 2 tbtrga kTkktk xxa +=



1. Initialize , i=0 and set2. Calculate and 3. Update network parameters

4. Calculate 5. Update

6. If halting condition hold, exitotherwise go to step 2

LM methodLM methodiθ λ

iθΔ

iii θθθ Δ+=+1

iαλ

LM.m



Calculate Calculate delta_thetadelta_thetaUpdate thetaUpdate theta



Control Control

Actual cost reduction

Predicted cost reduction

Current parameterNext parameter



ProjectProject

Learning RBF or MLP by the LM method1. (30 pt) Derivation

2. (140 pt) Matlab codes3. (50 pt) Testing

?)|(=

θθ

dtdy




],...,[] w w w w [

12*1

M210M2121

++==

MdM

TTM

TT

θθσσσθ LLLuuu

T

MdMdtdy

dtdy

dtdy ])|(,,)|([)|(

12*1 ++

=θ

θθθ

θθ

L




?)|(

?)|(

?)|(

=

=

=

m

m

m

dwtdy

dtdy

dtdy

θσθ

θu



MatlabMatlab codingcoding

• (30 pt)Main program

• Matlab Functions a. (10 pt)Calculate

b. (10 pt)Calculate )(L ii θ



MatlabMatlab codingcoding• Matlab functions

•(10 pt) Calculate



i

mdtdy

θθ

θ=|)|(

u

i

mdtdy

θθσθ

=|)|(

i

mdwtdy

θθ

θ=|)|(



MatlabMatlab codingcoding

•Matlab functions




)( iθ∇

)( iR θ

)|( θxG



TestTest•Give two examples to test your

matlab codes for learning RBF or MLP networks by Levenberg-Marquardt method

Levenberg-Marquardt learning @BULLETMATLAB programming @BULLET MLP (Multilayer Perceptrons)

Documents

Transcript of Levenberg-Marquardt learning @BULLETMATLAB programming @BULLET MLP (Multilayer Perceptrons)