hadoop培訓(xùn)1高級(jí)開發(fā)實(shí)戰(zhàn)

目前流行什么 +大數(shù)Hadoop1.0 Hadoop1.0HDFS分布SecondaryNameNode根據(jù)過editslogHadoop2.0Hadoop2.0新特性:NameNodeHadoop2.0HA Hadoop2.0新特性:通過 HadoopNFSv3實(shí)現(xiàn)方

NFS允許用戶像本地文件系統(tǒng) 通過引入一個(gè)NFSgateway服務(wù)實(shí)，該服務(wù)能將NFS協(xié)議轉(zhuǎn)換 協(xié)unnameduser(fileunnamedgroup(filenamednamed權(quán)限掩碼，用于過濾nameduser和namedgroup的權(quán)限MapReduceMapReduceMapReduceMapReduce

(2)assign

local

getmap

(onlocaldisks)

MapReduceMapReduceReduceMapper抽象1、protectedvoidsetup(Context2、protectedvoidmap(KEYINkey,VALUEINContextcontext)throwsIOException,InterruptedException{context.write((KEYOUT)key,(VALUEOUT)}3、protectedvoidcleanup(Context)throwsIOException,InterruptedException//}4、publicvoidrun(Contextcontext)throwsIOException,InterruptedException{while(context.nextKeyValue())}Reducer抽象1protectedvoidsetup(Context)throwsIOException,InterruptedException}2、protectedvoidreduce(KEYINkey, ble<VALUEIN>values,Context)throwsIOException,InterruptedException{for(VALUEINvalue:values){context.write((KEYOUT)key,(VALUEOUT)}}3、protectedvoidcleanup(Context)throwsIOException,InterruptedException}4、publicvoidrun(ContextcontextthrowsIOExceptionInterruptedExceptionwhile(context.nextKey())}} 文件輸入格式Defaultformat;readsoftextThebyteoffsetofTheParseslinesintokey-valEverythinguptothefirsttabcharacterofthelineAHadoop-specifichigh-performancebinaryformat輸入數(shù)據(jù)分塊輸入數(shù)據(jù)分塊InputSplit定義了輸入到單個(gè)任務(wù)的輸入數(shù)一個(gè)MapReduce程序被統(tǒng)稱InputSplit將文件分為64MB數(shù)據(jù)記錄讀入 讀入一個(gè)文本行數(shù)據(jù)記合并相同key的鍵值對(duì)，減少partitioner時(shí)候的數(shù)據(jù)通信開是在本地執(zhí)行的一個(gè)Reducer，滿足一定的條件才能夠執(zhí)行Partitioner&Partitioner&，用來決定一個(gè)給定(key,value)對(duì)傳給哪個(gè)節(jié)

傳輸?shù)矫恳粋€(gè)Reducer節(jié)點(diǎn)上新版本的編程接口是文件輸出格式 個(gè)號(hào)（partitionid） Default;writeslinesin"key\tvalue"WritesbinaryfilessuitableforreadingsubsequentMapReduceDisregardsitsHadoopStreaming介 運(yùn)行一個(gè)MapReduce運(yùn)行一個(gè)MapReduceStreamingcattest.log|wcfs-三、運(yùn)行/bin/hadoopjarcontrib/streaming/hadoop-streaming--mappercat-reducer'wc-- /data/test.log-outputHadoopMRhadoops MapReduce案例實(shí) Hadoop1.0的局限-Hadoop1.0的局限-擴(kuò)展可用

map-reducejob非常多的時(shí)候，會(huì)造成很大的內(nèi)存開銷，潛在來說，也增加了JobTrackerfail的風(fēng)險(xiǎn)，這也是業(yè)界普遍總結(jié)出老Hadoop的Map-Reduce只能支持4000節(jié)點(diǎn)主機(jī)的上限。存在單點(diǎn)故障,一旦故障，批處理模低效的資源管把資源強(qiáng)制劃分為maptaskslot和reducetaskslot,有maptask或者只有reducetask的時(shí)候，會(huì)造成資源的浪費(fèi)Hadoop2.0新特性:YARN

YARNHadoop2.0啟動(dòng) 處理來自 Hadoop2.0新特性:YARN-1.0資源調(diào)概概念:SlotHadoop1.0采用了靜態(tài)資源設(shè)置策略，即每個(gè)節(jié)點(diǎn)實(shí)現(xiàn)配Hadoop1.0將slot分為Mapslot和Reduceslot兩種，不允許共沒引入有效的資源機(jī)制，采用了基于jvm的資源機(jī)Hadoop2.0新特性:YARN的資源概概念:Container,實(shí)現(xiàn)資源動(dòng)態(tài)管理Hadoop2.0新特性:YARN的資源可