與Deep CNNs共享ImageNet分類的細(xì)節(jié)

SharingdetailofImageNetClassificationwithDeepCNNs林木得OutlineOverviewGoalDatasetModelMotivationArchitectureResultsPartIBasicProblemsActivationFunctionLossFunctionLearningMethodPartIIModelFeaturesReLUNonlinearityTrainingonMultipleGPUsLocalResponseNormalizationOverlappoolingReduceOverfittingDataAugmentationDropoutPartIIIMainphasesPreprocessInitializationStochasticgradientdescentTestReferencesOverviewGoalDatasetModelResultsGoalImageclassificationClassify

theImageNetLSVRC-2010contestimagesinto1000differentclasses.DataSetroughly1.2milliontrainingimages50,000validationimages150,000testingimagesModelMotivation利用自然圖像性質(zhì)

stationarity

of

statistics

locality

of

pixel

independencies模擬神經(jīng)網(wǎng)絡(luò)工作機(jī)理

receptivefieldModelArchitectureResultsTesterrorinILSVR-2010testsetResultsTesterrorinILSVR-2012testsetsPartIBasicProblemsActivationFunctionLostFunctionLearningMethodActivationFunctionForalllayersexceptoutputlayer: RectifiedLinearUnit(ReLU)TobeconfirmedForoutputlayer:

ReLUandLossFunctionmultinomiallogisticregressionobjective:

tobeconfirmed

LearningMethodGradientDescentTobemorespecific,StochasticGradientDescentwithbatchof128images.PartIIModelFeaturesReLUNonlinearityTrainingonMultipleGPUsLocalResponseNormalizationOverlappoolingReduceOverfittingDataAugmentationDropoutReLUNonlinearityStandardactivationfunction:f(x)=tanh(x)orf(x)=(1+ex)-1

Newinthispaper:

RectifiedLinearUnit(ReLU):

f(x)=max(0,x)

CIFAR-10PerformancecompariseTrainingonMultipleGPUsputshalfofthekernels(orneurons)oneachGPUtheGPUscommunicateonlyincertainlayers.readfromandwritetooneanother’smemorydirectly,Withouthostmachinememoryreducesourtop-1andtop-5errorratesby1.7%and1.2%LocalResponseNormalizationOnvalidationset

k=2,n=5,alpha=10-4,andbeta=0.75

In

realneurons,

橫向抑制reducesourtop-1andtop-5errorratesby1.4%and1.2%,respectively.OverlappoolingTraditionally,

non-overlappoolingNewinthispaper:Overlappoolings=2andz=3.educesthetop-1andtop-5errorratesby0.4%and0.3%,respectivelyWhypooling:

1,reducenumberofneuron 2,translateinvarianceOverallarchitectureOverallArchitectureNeuronineachlayers:224x224x3,55x55x96,27x27x256,13x13x394,13x13x394,13x13x256,4096,4096,1000.Almost:650,000neuronsParameterineachlayers:11x11x3x96,5x5x48x256,3x3x256x384,3x3x192x384,3x3x192x256,43264x4096,4096x4096,4096x1000Almost:60millionparametersReduceOverfittingReduceoverfittingisthemostimportantproblemforthismodelDataArgumentationgeneratingimagetranslationsandhorizontalreflec-tions.Train:Afactorof2048moreimagesTest:5x2imagesaveragepredictalteringtheintensitiesoftheRGBchannelsintrainingimages.toeachRGBimagepixelIxy=[IR,IG,IB]Tweaddthefollowingquantity:xyxyxyreducesthetop-1errorratebyover1%.

ReduceOverfittingDropoutMotivation:

Tooexpensivetocombinemanyabovemodelsthattakes5daystotrain

ReduceOverfittingDropoutHOW:

train:settingtozerotheoutputofeachhiddenneuronwithprobability0.5inthefirst2fully-connectlayers.

test:usealltheneuronsbutmultiplytheiroutputsby0.5ReduceOverfittingDropoutCost:

roughlydoublesthenumberofiterationsrequiredtoconverge

PartIIIMainphasesPreprocessInitializationStochasticgradientdescentTestPreprocessdown-sampledtheimagestoafixedresolutionof256x256rescaledtheimagesuchthattheshortersidewasoflength256croppedoutthecentral256x256patchfromtheresultingimagesubtractingthemeanactivityoverthetrainingsetfromeachpixel.Thustrainnetworkonthe(centered)rawRGBvaluesofthepixels.Initializationinitializedtheweightsineachlayerfromazero-meanGaussiandistributionwithstandardde-viation0.01.initializedtheneuronbiasesinthesecond,fourth,andfifth

convolutionallayers,aswellasinthefully-connectedhiddenlayers,withtheconstant1

initializedtheneuronbiasesintheremaininglayerswiththeconstant0learningratewasinitializedat0.01Stochasticgradientdescentwithabatchsizeof128examplesdecayof0.0005Updaterulesdividethelearningrateby10whenthevalidationerrorratestoppedimprovingwiththecurrentlearningrate.learningratereducedthreetimespriortotermination90cyclesthrough1.2millionimages

,took5to6daysTestAttesttime,thenetworkmakesapredictionbyextracting5x2224x224patchesaswellastheirhorizontalreflections(hencetenpatchesinall),andaveragingthepredictionsmadebythenetwork’ssoftmaxlayeronthetenpatches.Attesttime,weusealltheneuronsbutmultiplytheiroutputsby0.5

inthefirsttwofully-connectedlayers.References1,ImageNetClassifi