深度學(xué)習(xí)－－手寫(xiě)數(shù)字識(shí)別＜二＞－－交叉熵?fù)p失函數(shù)(crossentropycostfunc

1、深度學(xué)習(xí)一一手寫(xiě)數(shù)字識(shí)別二交叉熵?fù)p失函數(shù)(crossentropycostfunc。使用正確的代價(jià)函數(shù)避免學(xué)習(xí)減速交叉熵?fù)p失函數(shù)crossentropycostfuneton當(dāng)激活函數(shù)采用sigmoid()函數(shù)，損失函數(shù)使用二次和成本函數(shù)時(shí)：其中：此時(shí)輸出趨近于1且變化緩慢，即輸出對(duì)weights和biases的偏導(dǎo)的值非常小，由weights和biases的更新公式可以看出:此時(shí)weights和biases更新速度非常緩慢為解決神經(jīng)網(wǎng)絡(luò)學(xué)習(xí)慢的問(wèn)題，引入交叉熵?fù)p失函數(shù)(crossentropycostfunction)可以看出函數(shù)值大于等于0且當(dāng)a=y時(shí)，cost=O;因此符合作為costf

2、unction的條件.且可求得：可以看出，此時(shí)學(xué)習(xí)的速度取決于：因此，當(dāng)偏差越大時(shí)學(xué)習(xí)較快，偏差小時(shí)學(xué)習(xí)較慢最大似然損失函數(shù)log-likelyhood：costfunctionsoft函數(shù)：softmax是另為一種輸出層方程：神經(jīng)元節(jié)點(diǎn)的帶權(quán)輸入Z為：softmax輸出為：softmax輸出值都是大于等于0,且總和等于1,故可看作是概率分布.log-likelyhoodcostfunction：log-likelyhoodcostfunction即最大似然損失函數(shù).可以看出當(dāng)輸出比較接近目標(biāo)值時(shí)，概率a接近1,對(duì)數(shù)C接近0,反之概率較小時(shí)，對(duì)數(shù)C比較大.分別對(duì)weights和biases求偏

3、導(dǎo)：可以看出此時(shí)學(xué)習(xí)速度同樣取決與,因此不存在學(xué)習(xí)速度慢的問(wèn)題code2:#!/usr/bin/python#coding:utf-8importjsonimportrandomimportsysimportnumpyasnpimportcPickleimportgzipdefload_data():#讀取壓縮文件，返回一個(gè)描述符ff=gzip.open(./data/mnist.pkl.gz,rb)#從文件中讀取數(shù)據(jù)training_data,validation_data,test_data=cPickle.load(f)f.close()return(training_data,vali

4、dation_data,test_data)defload_data_wrapper():tr_d,va_d,te_d=load_data()#數(shù)據(jù)轉(zhuǎn)換tr_d是由50000個(gè)長(zhǎng)度為784的numpy.ndarray組成的tuple#轉(zhuǎn)換后的trainingnputs是由50000個(gè)長(zhǎng)度為784的numpy.ndarray組成的listtraining_inputs=np.reshape(x,(784,1)forxintr_d0training_results=vectorized_result(y)foryintr_d1訓(xùn)練集training_datazip()返回一個(gè)列表的元組，其中每個(gè)元

5、組包含從每個(gè)參數(shù)序列的第個(gè)元素。training_data=zip(training_inputs,training_results)validation_inputs=np.reshape(x,(784,1)forxinva_d0驗(yàn)證集validation_datavalidation_data=zip(validation_inputs,va_d1)test_inputs=np.reshape(x,(784,1)forxinte_d0測(cè)試集test_datatest_data=zip(test_inputs,te_d1)return(training_data,validation_dat

6、a,test_data)#當(dāng)預(yù)測(cè)值與目的值偏差較大時(shí)，會(huì)導(dǎo)致學(xué)習(xí)速度降低，故引/CrossEntropyCost=-sum(y*loga+(1-y)log(1-a)/n#交叉熵成本函數(shù)classCrossEntropyCost(object):staticmethoddeffn(a,y):#使用0代替數(shù)組x中的nan元素，使用有限的數(shù)字代替nf元素#sum(-ylog(a)-(1-y)log(1-a)returnnp.sum(np.nan_to_num(-y*np.log(a)-(1-y)*np.log(1-a)#返回從輸出層的誤差，注意參數(shù)Z不使用的方法staticmethoddefdelt

7、a(z,a,y):defdelta(z,a,y):return(a-y)#定義二次和成本函數(shù)classQuadraticCost(object):#返回與輸出相關(guān)聯(lián)的成本staticmethoddeffn(a,y):return0.5*np.linalg.norm(a-y)*2#返回從輸出層的誤差staticmethoddefdelta(z,a,y):return(a-y)*sigmoid_prime(z)classNetwork(object):def_init_(self,sizes,cost=CrossEntropyCost):#獲取神經(jīng)網(wǎng)絡(luò)的層數(shù)self.num_layers=len(

8、sizes)sizes即每層神經(jīng)元的個(gè)數(shù)self.sizes=sizes#賦隨機(jī)值(服從高斯分布)，對(duì)權(quán)重和偏向進(jìn)行初始化bais從第2行開(kāi)始self.default_weight_initializer()zip從傳入的可循環(huán)的兩組量中取出對(duì)應(yīng)數(shù)據(jù)組成一個(gè)tupleself.cost=cost#計(jì)算對(duì)應(yīng)的偏導(dǎo)數(shù)x-784維y-10維defbackprop(self,x,y):#返回一個(gè)元組(nabla_b,nabla_w)代表成本函數(shù)C_x的漸變。nabla_b和nabla_w是numpy數(shù)組np.array的逐層列表,類似于self.biases和self.weights.#分別生成與bi

9、asesweights等大小的0矩陣nabla_b=np.zeros(b.shape)forbinself.biasesnabla_w=np.zeros(w.shape)forwinself.weights#激活項(xiàng)直接傳入訓(xùn)練實(shí)例x的值activation=x#逐層存儲(chǔ)所有的激活(不止輸入層),作為一個(gè)列表activations=x#逐層存儲(chǔ)所有中間向量z,作為一個(gè)列表zs=forb,winzip(self.biases,self.weights):#計(jì)算中間變量Z=W*X+bz=np.dot(w,activation)+b#列表存儲(chǔ)所有中間向量zzs.append(z)#激活activati

10、on=sigmoid(W*X+b)activation=sigmoid(z)#列表存儲(chǔ)所有的激活activations.append(activation)#反向更新#輸出層計(jì)算輸出層error=Oj(1-Oj)(Tj-Oj);cost_derivative(activations-1,y)即C對(duì)a的梯度:(Tj-Oj)即(activations-1-y)sigmoid_prime(zs-1)即:Oj(1-Oj)delta=(self.cost).delta(zs-1,activations-1,y)#更新輸出層的nabla_b,nabla_wnabla_b-1=deltanabla_w-1=

11、np.dot(delta,activations-2.transpose()#隱藏層l=1表示神經(jīng)元的最后一層,l=2是第二層，依此類推反向更新直到初始層forlinxrange(2,self.num_layers):z=zs-lsp=sigmoid_prime(z)#weights-l+1即下一層的權(quán)重weights-l+1即下一層的權(quán)重delta=np.dot(self.weights-l+1.transpose(),delta)*sp#輸出C對(duì)w,b的偏導(dǎo)nabla_b-l=deltanabla_w-l=np.dot(delta,activations-l-1.transpose()re

12、turn(nabla_b,nabla_w)epochs訓(xùn)練多少輪,mini_batch_size抽取多少實(shí)例,eta學(xué)習(xí)率defSGD(self,training_data,epochs,mini_batch_size,eta,lmbda-正則化參數(shù)lmbda=0.0,evaluation_data=None,monitor_evaluation_cost=False,monitor_evaluation_accuracy=False,monitor_training_cost=False,monitor_training_accuracy=False):#驗(yàn)證集實(shí)例數(shù)量ifevaluatio

13、n_data:n_data=len(evaluation_data)n=len(training_data)evaluation_cost,evaluation_accuracy=,training_cost,training_accuracy=,#j代表第幾輪，共epochs輪forjinxrange(epochs):#將training_data中的數(shù)據(jù)隨機(jī)打亂random.shuffle(training_data)mini_batchs每次抽取mini_batch_size大小的數(shù)據(jù)作為一小塊，從0到n每次間隔mini_batch_size張圖片mini_batches=trainin

14、g_datak:k+mini_batch_sizeforkinxrange(0,n,mini_batch_size)#對(duì)取出來(lái)的mini_batchs逐個(gè)進(jìn)行更新formini_batchinmini_batches:#更新weights和biasesself.update_mini_batch(mini_batch,eta,lmbda,len(training_data)printEpoch%strainingcomplete%jifmonitor_training_cost:cost=self.total_cost(training_data,lmbda)training_cost.app

15、end(cost)printCostontrainingdata:.format(cost)ifmonitor_training_accuracy:accuracy=self.accuracy(training_data,convert=True)training_accuracy.append(accuracy)printAccuracyontrainingdata:/.format(accuracy,n)ifmonitor_evaluation_cost:#驗(yàn)證集損失cost=self.total_cost(evaluation_data,lmbda,convert=True)evalua

16、tion_cost.append(cost)printCostonevaluationdata:.format(cost)ifmonitor_evaluation_accuracy:#驗(yàn)證集準(zhǔn)確率accuracy=self.accuracy(evaluation_data)evaluation_accuracy.append(accuracy)printAccuracyonevaluationdata:/.format(accuracy,n_data)returnevaluation_cost,evaluation_accuracy,training_cost,training_accurac

17、yeta:學(xué)習(xí)率n:訓(xùn)練集實(shí)例數(shù)量#傳入單個(gè)的mini_batch,根據(jù)其x.y值，對(duì)整個(gè)神經(jīng)網(wǎng)絡(luò)的wights和biases進(jìn)行更新defupdate_mini_batch(self,mini_batch,eta,lmbda,n):#初始化nabla_b=np.zeros(b.shape)forbinself.biasesnabla_w=np.zeros(w.shape)forwinself.weightsforx,yinmini_batch:#計(jì)算對(duì)應(yīng)的偏導(dǎo)數(shù)delta_nabla_b,delta_nabla_w=self.backprop(x,y)delta_nabla_b,delta_

18、nabla_w=self.backprop(x,y)nabla_b=nb+dnbfornb,dnbinzip(nabla_b,delta_nabla_b)nabla_w=nw+dnwfornw,dnwinzip(nabla_w,delta_nabla_w)權(quán)重weights更新Wk=(1-(eta*lmbda/n)W-(eta/n)&C/&Wkself.weights=(1-eta*(lmbda/n)*w-(eta/len(mini_batch)*nwforw,nwinzip(self.weights,nabla_w)偏向biases更新bk=bk-(ets/n)&C/&bkself.bias

19、es=b-(eta/len(mini_batch)*nbforb,nbinzip(self.biases,nabla_b)#賦隨機(jī)值(服從標(biāo)準(zhǔn)正太分布)，對(duì)權(quán)重和偏向進(jìn)行初始化defdefault_weight_initializer(self):#第一層為輸入層不設(shè)置偏差，從第二行開(kāi)始self.biases=np.random.randn(y,1)foryinself.sizes1:#在同一個(gè)神經(jīng)元的權(quán)值的平方根的平方根上用高斯分布平均0和標(biāo)準(zhǔn)偏差1初始化每個(gè)權(quán)值self.weights=np.random.randn(y,x)/np.sqrt(x)forx,yinzip(self.size

20、s:-1,self.sizes1:)#使用平均0和標(biāo)準(zhǔn)差1的高斯分布初始化權(quán)重deflarge_weight_initializer(self):self.biases=np.random.randn(y,1)foryinself.sizes1:self.weights=np.random.randn(y,x)forx,yinzip(self.sizes:-1,self.sizes1:)#準(zhǔn)確率defaccuracy(self,data,convert=False)ifconvert:argmax()返回沿軸最大值的索引results=(np.argmax(self.feedforward(x

21、),np.argmax(y)for(x,y)indataelse:results=(np.argmax(self.feedforward(x),y)for(x,y)indatareturnsum(int(x=y)for(x,y)inresults)#如果數(shù)據(jù)集是訓(xùn)練集(通常情況)設(shè)置為false,驗(yàn)證集或測(cè)試集，則為truedeftotal_cost(self,data,lmbda,convert=False):cost=0.0forx,yindata:a=self.feedforward(x)ifconvert:y=vectorized_result(y)fn(a,y)=sum(-ylog(

22、a)-(1-y)log(1-a)則cost=fn(a,y)/ncost+=self.cost.fn(a,y)/len(data)#加上正則化項(xiàng)L2-regularization訓(xùn)練集包含實(shí)例個(gè)數(shù)：len(data)正則化項(xiàng)(lmbda/2n)*sum(w*2)cost+=0.5*(lmbda/len(data)*sum(np.linalg.norm(w)*2forwinself.weights)returncost#根據(jù)當(dāng)前輸入利用sigmoid函數(shù)來(lái)計(jì)算輸出deffeedforward(self,a):forb,winzip(self.biases,self.weights):a=sigmo

23、id(np.dot(w,a)+b)returna#保存神經(jīng)網(wǎng)絡(luò)文件filenamedefsave(self,filename):data=sizes:self.sizes,weights:w.tolist()forwinself.weights,biases:b.tolist()forbinself.biases,cost:str(self.cost._name_)f=open(filename,w)json.dump(data,f)f.close()#sigmoid函數(shù)defsigmoid(z):return1.0/(1.0+np.exp(-z)#sigmoid函數(shù)的導(dǎo)數(shù)#sigmoid函數(shù)

24、的導(dǎo)數(shù)defsigmoid_prime(z):returnsigmoid(z)*(1-sigmoid(z)#向量化defvectorized_result(j):e=np.zeros(10,1)ej=1.0returne#從filename加載神經(jīng)網(wǎng)絡(luò)，返回一個(gè)神經(jīng)網(wǎng)絡(luò)實(shí)例defload(filename):f=open(filename,r)data=json.load(f)f.close()cost=getattr(sys.modules_name_,datacost)net=Network(datasizes,cost=cost)net.weights=np.array(w)forwin

25、dataweightsnet.biases=np.array(b)forbindatabiasesreturnnetif_name_=_main_:training_data,valivation_data,test_data=load_data_wrapper()#顯示圖像ShowImage()static_foonet=Network(784,30,10)#訓(xùn)練集training_data,訓(xùn)練10輪，每次取樣10個(gè)作為mini_batch,學(xué)習(xí)率為3net.large_weight_initializer()用cross-entropy來(lái)識(shí)別MNIST數(shù)字times=400evaluation_cost,evaluation_accuracy,training_cost,training_accuracy=net.SG