人工神經(jīng)網(wǎng)絡(luò)-PPT課件

1、人工神經(jīng)網(wǎng)絡(luò)中國科學院自動化研究所吳高巍2016-11-29聯(lián)結(jié)主義學派 又稱仿生學派或生理學派認為人的思維基元是神經(jīng)元，而不是符號處理過程認為人腦不同于電腦核心：智能的本質(zhì)是聯(lián)接機制。原理：神經(jīng)網(wǎng)絡(luò)及神經(jīng)網(wǎng)絡(luò)間的連接機制和學習算法麥卡洛可（McCulloch）皮茨（Pitts)什么是神經(jīng)網(wǎng)絡(luò)所謂的人工神經(jīng)網(wǎng)絡(luò)就是基于模仿生物大腦的結(jié)構(gòu)和功能而構(gòu)成的一種信息處理系統(tǒng)（計算機）。個體單元相互連接形成多種類型結(jié)構(gòu)的圖循環(huán)、非循環(huán)有向、無向自底向上（Bottom-Up）AI起源于生物神經(jīng)系統(tǒng)從結(jié)構(gòu)模擬到功能模擬仿生人工神經(jīng)網(wǎng)絡(luò)內(nèi)容生物學啟示多層神經(jīng)網(wǎng)絡(luò)Hopfield網(wǎng)絡(luò)自組織網(wǎng)絡(luò)生物學啟示 神經(jīng)元

2、組成：細胞體，軸突，樹突，突觸 神經(jīng)元之間通過突觸兩兩相連。信息的傳遞發(fā)生在突觸。 突觸記錄了神經(jīng)元間聯(lián)系的強弱。 只有達到一定的興奮程度，神經(jīng)元才向外界傳輸信息。 生物神經(jīng)元神經(jīng)元神經(jīng)元特性信息以預知的確定方向傳遞一個神經(jīng)元的樹突細胞體軸突突觸另一個神經(jīng)元樹突時空整合性對不同時間通過同一突觸傳入的信息具有時間整合功能對同一時間通過不同突觸傳入的信息具有空間整合功能神經(jīng)元工作狀態(tài)興奮狀態(tài)，對輸入信息整合后使細胞膜電位升高，當高于動作電位的閾值時，產(chǎn)生神經(jīng)沖動，并由軸突輸出。抑制狀態(tài)，對輸入信息整合后使細胞膜電位降低，當?shù)陀趧幼麟娢坏拈撝禃r，無神經(jīng)沖動產(chǎn)生。結(jié)構(gòu)的可塑性神經(jīng)元之間的柔性連接：突觸

3、的信息傳遞特性是可變的學習記憶的基礎(chǔ)神經(jīng)元模型從生物學結(jié)構(gòu)到數(shù)學模型人工神經(jīng)元M-P模型x1x2xny12nInputOutputThresholdMcClloch and Pitts, A logical calculus of the ideas immanent in nervous activity, 1943f: 激活函數(shù)(Activation Function)g: 組合函數(shù)(Combination Function)Weighted Sum Radial Distance組合函數(shù) (e) (f)ThresholdLinearSaturating LinearLogistic Si

4、gmoidHyperbolic tangent SigmoidGaussian激活函數(shù)人工神經(jīng)網(wǎng)絡(luò)多個人工神經(jīng)元按照特定的網(wǎng)絡(luò)結(jié)構(gòu)聯(lián)接在一起，就構(gòu)成了一個人工神經(jīng)網(wǎng)絡(luò)。神經(jīng)網(wǎng)絡(luò)的目標就是將輸入轉(zhuǎn)換成有意義的輸出。生物系統(tǒng)中的學習自適應學習適應的目標是基于對環(huán)境信息的響應獲得更好的狀態(tài)在神經(jīng)層面上，通過突觸強度的改變實現(xiàn)學習消除某些突觸，建立一些新的突觸生物系統(tǒng)中的學習Hebb學習律神經(jīng)元同時激活，突觸強度增加異步激活，突觸強度減弱學習律符合能量最小原則保持突觸強度需要能量，所以在需要的地方保持，在不需要的地方不保持。ANN的學習規(guī)則能量最小 ENERGY MINIMIZATION對人工神經(jīng)網(wǎng)絡(luò)

5、，需要確定合適的能量定義；可以使用數(shù)學上的優(yōu)化技術(shù)來發(fā)現(xiàn)如何改變神經(jīng)元間的聯(lián)接權(quán)重。ENERGY = measure of task performance error兩個主要問題結(jié)構(gòu) How to interconnect individual units?學習方法 How to automatically determine the connection weights or even structure of ANN?Solutions to these two problems leads to a concrete ANN!人工神經(jīng)網(wǎng)絡(luò)前饋結(jié)構(gòu)(Feedforward Archite

6、cture) - without loops - static 反饋/循環(huán)結(jié)構(gòu)(Feedback/Recurrent Architecture) - with loops - dynamic (non-linear dynamical systems)ANN結(jié)構(gòu)General structures of feedforward networksGeneral structures of feedback networks通過神經(jīng)網(wǎng)絡(luò)所在環(huán)境的模擬過程，調(diào)整網(wǎng)絡(luò)中的自由參數(shù) Learning by data學習模型 Incremental vs. Batch兩種類型 Supervised vs.

7、 UnsupervisedANN的學習方法若兩端的神經(jīng)元同時激活，增強聯(lián)接權(quán)重Unsupervised Learning學習策略: Hebbrian Learning 最小化實際輸出與期望輸出之間的誤差(Supervised) - Delta Rule (LMS Rule, Widrow-Hoff) - B-P LearningObjective:Solution:學習策略: Error Correction采用隨機模式，跳出局部極小 - 如果網(wǎng)絡(luò)性能提高，新參數(shù)被接受. - 否則，新參數(shù)依概率接受Local MinimumGlobal Minimum學習策略: Stochastic Lear

8、ning“勝者為王”(Winner-take-all )UnsupervisedHow to compete? - Hard competition Only one neuron is activated - Soft competition Neurons neighboring the true winner are activated. 學習策略: Competitive Learning重要的人工神經(jīng)網(wǎng)絡(luò)模型多層神經(jīng)網(wǎng)絡(luò)徑向基網(wǎng)絡(luò)Hopfield網(wǎng)絡(luò)Boltzmann機自組織網(wǎng)絡(luò)多層感知機(MLP)感知機實質(zhì)上是一種神經(jīng)元模型閾值激活函數(shù)Rosenblatt, 1957感知機判別規(guī)則

9、輸入空間中樣本是空間中的一個點權(quán)向量是一個超平面超平面一邊對應 Y=1另一邊對應 Y=-1單層感知機學習調(diào)整權(quán)值，減少訓練集上的誤差簡單的權(quán)值更新規(guī)則:初始化對每一個訓練樣本:Classify with current weightsIf correct, no change!If wrong: adjust the weight vector30學習: Binary Perceptron初始化對每一個訓練樣本:Classify with current weightsIf correct (i.e., y=y*), no change!If wrong: adjust the weight

10、vector by adding or subtracting the feature vector. Subtract if y* is -1.多類判別情況If we have multiple classes:A weight vector for each class:Score (activation) of a class y:Prediction highest score wins學習: Multiclass Perceptron初始化依次處理每個樣本Predict with current weightsIf correct, no change!If wrong: lower

11、 score of wrong answer, raise score of right answer感知機特性可分性: true if some parameters get the training set perfectly correctCan represent AND, OR, NOT, etc., but not XOR收斂性: if the training is separable, perceptron will eventually converge (binary case)SeparableNon-Separable感知機存在的問題噪聲（不可分情況）: if the

12、data isnt separable, weights might thrash泛化性: finds a “barely” separating solution改進感知機線性可分情況Which of these linear separators is optimal? Support Vector MachinesMaximizing the margin: good according to intuition, theory, practiceOnly support vectors matter; other training examples are ignorable Supp

13、ort vector machines (SVMs) find the separator with max marginSVM優(yōu)化學習問題描述訓練數(shù)據(jù)目標：發(fā)現(xiàn)最好的權(quán)值，使得對每一個樣本x的輸出都符合類別標簽樣本xi的標簽可等價于標簽向量采用不同的激活函數(shù)平方損失：單層感知機單層感知機單層感知機單層感知機采用線性激活函數(shù)，權(quán)值向量具有解析解批處理模式一次性更新權(quán)重缺點：收斂慢增量模式逐樣本更新權(quán)值隨機近似，但速度快并能保證收斂多層感知機 (MLP)層間神經(jīng)元全連接MLPs表達能力3 layers: All continuous functions 4 layers: all functio

14、nsHow to learn the weights?waiting B-P algorithm until 1986B-P Network結(jié)構(gòu) A kind of multi-layer perceptron, in which the Sigmoid activation function is used.B-P 算法學習方法 - Input data was put forward from input layer to hidden layer, then to out layer - Error information was propagated backward from out

15、 layer to hidder layer, then to input layerRumelhart & Meclelland, Nature,1986B-P 算法Global Error Measuredesired outputgenerated outputsquared errorThe objective is to minimize the squared error, i.e. reach the Minimum Squared Error (MSE)B-P 算法Step1. Select a pattern from the training set and present

16、 it to the network.Step2. Compute activation of input, hidden and output neurons in that sequence.Step3. Compute the error over the output neurons by comparing the generated outputs with the desired outputs.Step4. Use the calculated error to update all weights in the network, such that a global erro

17、r measure gets reduced. Step5. Repeat Step1 through Step4 until the global error falls below a predefined threshold.梯度下降方法Optimization method for finding out the weight vector leading to the MSE learning rategradientvector form:element form:權(quán)值更新規(guī)則For output layer:權(quán)值更新規(guī)則For output layer:權(quán)值更新規(guī)則For hid

18、den layer權(quán)值更新規(guī)則For hidden layer應用: Handwritten digit recognition3-nearest-neighbor = 2.4% error40030010 unit MLP = 1.6% errorLeNet: 768 192 30 10 unit MLP = 0.9% errorCurrent best (SVMs) 0.4% errorMLPs：討論實際應用中Preprocessing is importantNormalize each dimension of data to -1, 1Adapting the learning ra

19、tet = 1/tMLPs：討論優(yōu)點：很強的表達能力容易執(zhí)行缺點：收斂速度慢過擬合（Over-fitting）局部極小采用Newton法加正則化項，約束權(quán)值的平滑性采用更少（但足夠數(shù)量）的隱層神經(jīng)元嘗試不同的初始化增加擾動 Hopfield 網(wǎng)絡(luò)反饋 結(jié)構(gòu)可用加權(quán)無向圖表示Dynamic System兩種類型 Discrete (1982) and Continuous (science, 1984), by HopfieldHopfield網(wǎng)絡(luò)Combination function：Weighted SumActivation function：Threshold吸引子與穩(wěn)定性How do

20、 we “program” the solutions of the problem into stable states (attractors) of the network?How do we ensure that the feedback system designed is stable? Lyapunovs modern stability theory allows us to investigate the stability problem by making use of a continuous scalar function of the state vector,

21、called a Lyapunov (Energy) Function.Hopfield網(wǎng)絡(luò)的能量函數(shù)With inputWithout inputHopfield 模型Hopfield證明了異步Hopfield網(wǎng)絡(luò)是穩(wěn)定的，其中權(quán)值定義為 Whatever be the initial state of the network, the energy decreases continuously with time until the system settles down into any local minimum of the energy surface.Hopfield 網(wǎng)絡(luò): 聯(lián)

22、想記憶Hopfield網(wǎng)絡(luò)的一個主要應用基于與數(shù)據(jù)部分相似的輸入，可以回想起數(shù)據(jù)本身（attractor state）也稱作內(nèi)容尋址記憶(content-addressable memory).Stored PatternMemory Association虞臺文, Feedback Networksand Associative MemoriesHopfield 網(wǎng)絡(luò): Associative MemoriesStored PatternMemory Association虞臺文, Feedback Networksand Associative MemoriesHopfield網(wǎng)絡(luò)的一個主

23、要應用基于與數(shù)據(jù)部分相似的輸入，可以回想起數(shù)據(jù)本身（attractor state）也稱作內(nèi)容尋址記憶(content-addressable memory).How to store patterns?=?How to store patterns?=?: Dimension of the stored pattern權(quán)值確定: 外積(Outer Product)Vector form: Element form:Why? Satisfy the Hopfield modelAn example of Hopfield memory 虞臺文, Feedback Networks and As

24、sociative Memories123422123422111111111111StableE=4E=0E=4Recall the first pattern (x1)123422111111111111StableE=4E=0E=4Recall the second pattern (x2)Hopfield 網(wǎng)絡(luò): 組合優(yōu)化(Combinatorial Optimization)Hopfield網(wǎng)絡(luò)的另一個主要應用將優(yōu)化目標函數(shù)轉(zhuǎn)換成能量函數(shù)(energy function)網(wǎng)絡(luò)的穩(wěn)定狀態(tài)是優(yōu)化問題的解例: Solve Traveling Salesman Problem (TSP)Gi

25、ven n cities with distances dij, what is the shortest tour?Illustration of TSP Graph1234567891011Hopfield Network for TSP=?Hopfield Network for TSP=City matrix Constraint 1. Each row can have only one neuron “on”. 2. Each column can have only one neuron “on”. 3. For a n-city problem, n neurons will

26、be on.Hopfield Network for TSP124351234512345TimeCityThe salesman reaches city 5 at time 3.Weight determination for TSP: Design Energy FunctionConstraint-1Constraint-2Constraint-3能量函數(shù)轉(zhuǎn)換為2DHopfield網(wǎng)絡(luò)形式Network is built!Hopfield網(wǎng)絡(luò)迭代（TSP ） The initial state generated randomly goes to the stable state (s

27、olution) with minimum energyA 4-city example 阮曉剛， 神經(jīng)計算科學，2006自組織特征映射 (SOFM) What is SOFM?Neural Network with Unsupervised LearningDimensionality reduction concomitant with preservation of topological information. Three principals - Self-reinforcing - Competition - CooperationStructure of SOFM競爭(Competition)Finding the best matching weight vector for the present input.Criterion for determining the winning neuron: Maximum Inner Product Minimum Euclidean Distance合作(Cooperation