1、Convolution Neural NetworkCNNA tutorialKH WongConvolution Neural Network CNN ver. 4.11a1Introduction Very Popular: Toolboxes: cuda-convnet and caffe (user friendlier) A high performance Classifier (multi-class) Successful in handwritten optical character OCR recognition, speech recognition, image no

2、ise removal etc. Easy to implementation Slow in learning Fast in classificationConvolution Neural Network CNN ver. 4.11a2Overview of this note Part 1: Fully connected Back Propagation Neural Networks (BPNN) Part 1A: feed forward processing Part 1A: feed backward processing Part 2: Convolution neural

3、 networks (CNN) Part 2A: feed forward of CNN Part 2B: feed backward of CNNConvolution Neural Network CNN ver. 4.11a3Part 1Fully Connected Back Propagation (BP) neural netConvolution Neural Network CNN ver. 4.11a4TheoryFully connected Back Propagation Neural Net (BPNN) Use many samples to train the w

4、eights, so it can be used to classify an unknown input into different classes Will explain How to use it after training: forward pass How to train it: how to train the weights and biases (using forward and backward passes)Convolution Neural Network CNN ver. 4.11a5Training How to train it: how to tra

5、in the weights (W) and biases (b) (use forward, backward passes) Initialize W and b randomly Iter=1: all_epocks (each is called an epcok) Forward pass for each output neuron: Use training samples: Xclass_t : feed forward to find y. Err=error_function(y-t) Backward pass: Find W and b to reduce Err. W

6、new=Wold+W; bnew=bold+bConvolution Neural Network CNN ver. 4.11a6Part 1AForward pass of Back Propagation Neural Net (BPNN)Recall:Forward pass for each output neuron:-Use training samples: Xclass_t : feed forward to find y.-Err=error_function(y-t)Convolution Neural Network CNN ver. 4.11a7Feed forward

7、 of Back Propagation Neural Net (BPNN) In side each neuron: Convolution Neural Network CNN ver. 4.11a81xl2xl3xl2lw3lwNlwNlx)b(11)u(x,11)(i.e. function, (sigmod) logistic a is Typicallyb,W,u,xsuch that ,bwith )u(xlllxWllullllllllllllllefthereforeeuf fbWuxxWufOutput neuronsInputsSigmod function f(u) a

8、nd its derivative f(u) Convolution Neural Network CNN ver. 4.11a9)(1)()(1 slope for theparamter ,simplicityFor )(1)()1 ()1 (1)1 ()1 ()()()1 (11)()(slopefor paramter theis ,11)(22ufufufufufeeeeeeeufduedfededduudfufHenceeufuuuuuuuuuuuhttp:/ single neuron The neural net can have many layers In between

9、any neighboring 2 layers, a set of neurons can be foundConvolution Neural Network CNN ver. 4.11a10llllllbxWuufx1 with)(W,Wx,1layer at inputs 1wweightsxlxlllxllayer at input ) 1 (1lxlx) 1 (lW)2(lW)2(1lxlu lufEach NeuronBPNN Forward pass Forward pass is to find output when an input is given. For examp

10、le: Assume we have used N=60,000 images to train a network to recognize c=10 numerals. When an unknown image is input, the output neuron corresponds to the correct answer will give the highest output level.Convolution Neural Network CNN ver. 4.11a1110 output neurons for 0,1,2,.,9InputimageThe criter

11、ia to train a network Is based on the overall error function Convolution Neural Network CNN ver. 4.11a12network forward feed theofouput at the sample training theof classoutput The sample training theof class truegiven The;2yt2121:neuron eachfor Error 21error Overall2221211thnkthnknncknknknNncknknkN

12、nyntnormsytEytEStructure of a BP neural network Convolution Neural Network CNN ver. 4.11a13Input layeroutput layerllllllllbWuxb,W,u,x that suchbiases ofset b weights,ofset W inputs, ofset x1layer hidden lllayer hidden 1xllx()bWfbiasesweightsllArchitecture (exercise: write formulas for A1(i=4) and A2

13、(k=3)Convolution Neural Network CNN ver. 4.11a Input:P=9x1Indexed by j Hidden layer =5 neurons,indexed by iW1=9x5b1=5x1W1(j=1,i=1)W1(j=2,i=1)W1(j=9,i=1)P(j=1)P(j=2)P(j=3):P(j=9)1(ib.)1, 2()1, 1(11211111APijWPijWeA1(i=1)P(j=1)P(j=2)P(j=9)Neuron i=1Bias=b1(i=1)W2(i=1,k=1)W2(i=2,k=1)W2(i=5,k=1)1(b.)1()

14、1, 2()1()1, 1(22221211AkkAkiWkAkiWeA2(k=2)A1A2A5Neuron k=1Bias=b2(k=1)W1(j=1,i=1)W1(j=2,i=1)W1(j=9,i=5)W1(j=3,i=4)A1(i=5)A1(i=1)Output neurons=3 neurons,indexed by kW2=5x3b2=3x1W2(i=5,k=3)W2(i=1,k=1)W2(i=2,k=2)W2(i=2,k=1)A1(i=2)14Answer (exercise: write values for A1(i=4) and A2(k=3) P= 0.7656 0.734

15、4 0.9609 0.9961 0.9141 0.9063 0.0977 0.0938 0.0859 W1= 0.2112 0.1540 -0.0687 -0.0289 0.0720 -0.1666 0.2938 -0.0169 -0.1127 -b1= 0.1441 %Find A1(i=4) A1_i_is_4=1/(1+exp-(W1*P+b1) =0.49Convolution Neural Network CNN ver. 4.11a15)1(ib.)4, 2()4, 1(11211111)4(APijWPijWeiNumerical example for the forward

16、path Feed forward Give numbers of x, w b etcConvolution Neural Network CNN ver. 4.11a16Example: a simple BPNN Number of classes (no. of output neurons)=3 Input 9 pixels: each input is a 3x3 image Training samples =3 for each class Number of hidden layers =1 Number of neurons in the hidden layer =5 C

17、onvolution Neural Network CNN ver. 4.11a17Architecture of the exampleConvolution Neural Network CNN ver. 4.11a18Input Layer9x1 pixelsoutput Layer 3x1 5x1b5x9Wlayer hidden xlx()bWfbiasesweightsll Part 1BBackward pass of Back Propagation Neural Net (BPNN)Convolution Neural Network CNN ver. 4.11a19feed

18、back Convolution Neural Network CNN ver. 4.11a201ll1lxlxFeedforwardFeedbackwardllayer )(1bwxfxl lllTlllTllffWfWu1uu1111 Convolution Neural Network CNN ver. 4.11a21 nnLLnnnlnnnnnnlnnnnnlnnnnnnlllltyfLivuftybEbuibuufytbuftytbEbytytbEtufyiiiuftytEiibuubhenceibubxWuulayer output at the)(, 1),(in since,)

19、( (iii), & (ii) Fromor target truth theis ,outputcurrent theis )( since,)()(2121 sampleth -n theSince)(essensitivit theEE),(1 so, since221derivationderivation Convolution Neural Network CNN ver. 4.11a22 llllllTllnnlnnlnnlnnlnnnlnnnWWWWWWuftyxuftyWbwxuftyWufytWyytWytEEhencefactor learninga useslo

20、wly it do to,E make so negative makeclcycle learningeverfy for W decease want to weif ,W Wcalculated isW new a phase, learning eachFor x)( (iv) in since,)( )( )(E21 , (iii) from Alsooldnew12Numerical example for the feed back passConvolution Neural Network CNN ver. 4.11a23Procedure From the last lay

21、er (output), find dt-y Find d, then find w of the whole network Find iterative (forward- back forward pass) to generate a new set of W, until dW is small Takes a long timeConvolution Neural Network CNN ver. 4.11a24Part 2Convolution Neural NetworksPart 2AFeed forward part of cnnff( )Convolution Neura

22、l Network CNN ver. 4.11a25Matlab examplehttp:/ example optical chartered recognition OCRExample test_example_CNN.m in http:/ on a data base (mnist_uint8, from http:/ training examples (28x28 pixels each)10,000 testing samples (a different dat.2set) After training , given an unknown image, it will te

23、ll whether it is 0, or 1 ,.,9 etc. Recognition rate 11% use 1 epoch (training 200seconds) Recognition rate 1.2% use 100 epochs (hours of training) Convolution Neural Network CNN ver. 4.11a26http:/ ofTest_example_CNN.mRead data basePart I: cnnsetup.m Layer 1: input layer (do nothing) Layer 2 convolut

24、ion(conv.) Layer, output maps=6, kernel size=5x5 Layer 3 sub-sample (subs.) Layer, scale=2 Layer 4 conv. Layer, output maps =12, kernel size=5x5 Layer 5 subs. Layer (output layer), scale =2Part 2: cnntrain.m % train wedihgts using 60,000 samples cnnff( ) % CNN feed forward cnndb( ) % CNN feed back t

25、o train weighted in kernels cnnapplygrads( ) % update weightscnntest.m % test the system using 10000 samples and show error rate Convolution Neural Network CNN ver. 4.11a27Architecture Convolution Neural Network CNN ver. 4.11a28Each output neuron corresponds to a character (0,1,2,.,9 etc.)Layer 1:Im

26、age Input1x28x28Layer 12:6 conv.Maps (C)InputMaps=6OutputMaps=6Fan_in=52=25 Fan_out=6x52=150Layer 23:6 sub-sample Map (S)InputMaps=6OutputMaps=12Layer 34:12 conv.Maps (C)InputMaps=6OutputMaps=12Fan_in=6x52=150Fan_out=12x52=300Layer 45:12 sub-sample Map (S)InputMaps=12OutputMaps=12Layer 1:One input (

27、I)Layer 5:12x4x410Kernel=5x52x2Kernel=5x5SubsSubsLayer 4:12x8x8Layer 3:6x12x12Layer 2:6x24x24Conv.I=inputC=Conv.=convolutionS=Subs=sub samplingConv.2x2Cnnff.mconvolution neural networks feed forward This is the feed forward part Assume all the weights are initialized or calculated, we show how to ge

28、t the output from inputs.Convolution Neural Network CNN ver. 4.11a29Layer 12:Convolute layer 1 with different kernels (map_index1=1,2,.,6) and produce 6 output mapsInputs : input layer 1, a 28x28 image6 different kernels : k(1),.,k(6) , each k is 5x5, K are dendrites of neurons Output : 6 output map

29、s each 24x24AlgorithmFor(map_index=1:6)layer_2(map_index)= I*k(map_index)validDiscussionValid means only consider overlapped areas, so if layer 1 is 28x28, kernel is 5x5 each, each output map is 24x24 In Matlab use convn(I,k,valid)Example:I=rand(28,28)k=rand(5,5)size(convn(I,k,valid) ans 24 24Convol

30、ution Neural Network CNN ver. 4.11a30Layer 1:Image Input (i)1x28x28Layer 12:6 conv.Maps (C)InputMaps=6OutputMaps=6Fan_in=52=25 Fan_out=6x52=150Layer 1:One input (I)Kernel=5x52x2Layer 2(c):6x24x24Conv.*K(6)I=inputC=Conv.=convolutionS=Subs=sub samplingijMap_index=1 2 : 6Conv.*K(1)Layer 23: Convolution

31、 Neural Network CNN ver. 4.11a31Layer 23:6 sub-sample Map (S)InputMaps=6OutputMaps=122x2SubsLayer 3 (s):6x12x12Layer 2 (c):6x24x24Sub-sample layer 2 to layer 3Inputs : 6 maps of layer 2, each is 24x24Output : 6 maps of layer 3, each is 12 x12AlgorithmFor(map_index=1:6)For each input map, calculate t

32、he average of 2x2 pixels and the result is saved in output maps.Hence resolution is reduced from 24x24 to 12x12DiscussionMap_index=1 2 : 6Layer 34:Convolution Neural Network CNN ver. 4.11a32Layer 34:12 conv.Maps (C)InputMaps=6OutputMaps=12Fan_in=6x52=150Fan_out=12x52=300Kernel=5x5Layer 4(c):12x8x8La

33、yer3 L3(s):6x12x12 Conv. layer 3 with kernels to produce layer 4Inputs : 6 maps of layer3(L3i=1:6), each is 12x12Kernel set: totally 6x12 kernels, each is 5x5,i.e.Ki=1:6j=1:12, each Kij is 5x512 biasj=1:12 in this layer, each is a scalarOutput : 12 maps of layer4(L4j=1:12), each is 8x8Algorithmfor(j

34、=1:12) for (i=1:6) clear z, i.e. z=0; z=z+covn (L3i, kij,valid) %z is 8x8 L4j=sigm(z+baisj) %L4j is 8x8 function X = sigm(P) X = 1./(1+exp(-P);EndDiscussionNormalization?Index=i=1:6Index=j=1:12:net.layersl.ajLayer 45 Convolution Neural Network CNN ver. 4.11a33Layer 45:12 sub-sample Map (S)InputMaps=

35、12OutputMaps=12Layer 5:12x4x410SubsLayer 4:12x8x82x2 Subsample layer 4 to layer 5 Inputs : 12 maps of layer4(L4i=1:12), each is 12x8x8 Output : 12 maps of layer5(L5j=1:12), each is 4x4 Algorithm Sub sample each 2x2 pixel window in L4 to a pixel in L5 Discussion Normalization?Layer 5output Convolutio

36、n Neural Network CNN ver. 4.11a34Each output neuron corresponds to a character (0,1,2,.,9 etc.)net.om=1:10Layer 45:12 sub-sample Map (S)InputMaps=12OutputMaps=12Layer 5 (L5j=1:12:12x4x4=192Totally 192 pixels10Subsample layer 4 to layer 5Inputs : 12 maps of layer5(L5i=1:12), each is 4x4, so L5 has 19

37、2 pixels in totalOutput layer weights: Net.ffWm=1:10p=1:192, total number of weights is 192Output : 10 output neurons (net.om=1:10)AlgorithmFor m=1:10%each output neuronclear net.fvnet.fv=Net.ffWmall 192 weight.*L5(all corresponding 192 pixels)net.om=sign(net.fv + bias)Discussion: :Totally192 weight

38、s for each output neuronSame for each output neuronPart 2BBack propagation partcnnbp( )cnnapplyweight( )Convolution Neural Network CNN ver. 4.11a35cnnbp( )overview (output back to layer 5 Convolution Neural Network CNN ver. 4.11a36net.od) * (net.ffW net.fvdcnnbp.m codein so*net.o) - (1 * . (net.o *

39、. net.e*.net.o) - (1 * . (net.o * . net.e.1)1 ()()(._)1 ()(iiiiiiiiiiixEwwodnetxEodnetwxEwxyytyxEtyenetyooutmcnnbpinxyytywERef: See /wiki/BackpropagationLayer 5 to 4 Expand 1x1 to 2x2Convolution Neural Network CNN ver. 4.11a37Layer 4 to 3 Rotated convolution Find dE/dx at layer

40、 3Convolution Neural Network CNN ver. 4.11a38Layer 3 to 2 Expand 1x1 to 2x2Convolution Neural Network CNN ver. 4.11a39Calculate gradient From later 2 to layer 3 From later 3 to layer 4 Net.ffW Net.ffb foundConvolution Neural Network CNN ver. 4.11a40Details of calc gradients % part % reshape feature

41、vector deltas into output map style L4(c) run expand only L3(s) run conv (rot180, fill), found d L2(c) run expand only %Part % calc gradients L2(c) run conv (valid), found dk and db L3(s) not run here L4(c) run conv(valid), found dk and db Done , found these for the output layer L5: net.dffW = net.o

42、d * (net.fv) / size(net.od, 2); net.dffb = mean(net.od, 2);Convolution Neural Network CNN ver. 4.11a41cnnapplygrads(net, opts) For the convolution layers, L2, L4 From k and dk find new k (weights) From b and db find new b (bias) For the output layer L5 net.ffW = net.ffW - opts.alpha * net.dffW; net.ffb = net.ffb - opts.alpha * net.dffb; opts.alpha is to