云計(jì)算-Hadoop單機(jī)及集群配置

1、實(shí)驗(yàn)報(bào)告實(shí)驗(yàn)名稱Hadoop單機(jī)及集群配置指導(dǎo)教師實(shí)驗(yàn)類型上機(jī)實(shí)驗(yàn)學(xué)時(shí)2實(shí)驗(yàn)時(shí)間一、實(shí)驗(yàn)?zāi)康暮鸵髮?shí)驗(yàn)要求：學(xué)會(huì)Hadoop單機(jī)版及集群的安裝配置實(shí)驗(yàn)?zāi)康模?）了解Hadoop的安裝及運(yùn)行環(huán)境；2）掌握Hadoop的工作原理；3）在此基礎(chǔ)上為Hadoop數(shù)據(jù)分析奠定基礎(chǔ)二、實(shí)驗(yàn)環(huán)境在虛擬機(jī)上運(yùn)行ubuntu系統(tǒng)，hadoop-1.2.1三、實(shí)驗(yàn)原理及內(nèi)容安裝Ubuntuhadoop用戶組（1）創(chuàng)建hadoop用戶組：sudoaddgrouphadooppublicubuntu:$sudoaddgrouphadoopsudopasswordforpublic:Addinggrouphadoop(

2、GID1001)iDone” 創(chuàng)建hadoop用戶：sudoadduser-ingrouphadoophadooppubLi匚電ubuntusudoadduser-tngrouphadoophadoopAddinguserhadoop1*Addingnewuserhadoop1(1001)withgrouphadoop1Creatinghomedire匚to廠y/home/hadoop.Copyingfilesfrom/etc/skel1EnternewUNIXpassword:RetypenewUNIXpassword:p白passwordupdatedsuccessfullyChangin

3、gtheuserinforna七ionforhadoopEnterthenewvalue,orpressENTERforthedefauItFullNan?:RoomNumber;WorkPhone:HonePhoneiOther:Istheinformationcorrect?Y/ny(3)為hadoop用戶添加權(quán)限：sudogedit/etc/sudoers在打開的文件中添加一句hadoopALL二(ALL：ALL)ALL,其中hadoop和ALL之間的空格使用TAB鍵。#UserprivilegespeciftcotiongtALL=(ALL;ALL)ALLhadoopALL=(ALL：

4、ALL)ALL使用hadoop用戶登錄系統(tǒng)重新打開Ubuntu，發(fā)現(xiàn)登陸界面多了一個(gè)hadoop用戶，用hadoop用戶登錄系統(tǒng)。3、SSH配置Ubuntul4.04已默認(rèn)安裝ssh的客戶端，只需要安裝服務(wù)器端即可，但在實(shí)際操作時(shí)，發(fā)現(xiàn)由于是剛安裝的Ubuntu系統(tǒng)，該系統(tǒng)沒有更新，還沒有安裝ssh，因此先使用系統(tǒng)更新命令sudoapt-getupdate更新系統(tǒng)。ha)doopubuntu:$&udo-getupdate安裝ssh：sudoapt-getinstallopensshserver。hadoopubuntu:sudoaptgetinstallopenssh-serverj啟動(dòng)ss

5、h：sudo/etc/init.d/sshstart查看服務(wù)是否正確啟動(dòng)：pse|grepsshhsdoopubuntus5sudo/etc/init/礙hstarthadoopubuntur$ps-e|grepssh?00:&Or00ssfd設(shè)置免密碼登錄，生成私鑰和公鑰：sshkeygentrsaPhddoopubuntu:$ssh-keygen-trsa-P”Generatingpublic/privatersakeypair.Enterfileinwhichtosavethekey(/home/hodoop/*ssh/id_rsa);(5)將公鑰追加到authorized_keys(a

6、uthorized_keys用于保存所有允許以當(dāng)前用戶身份登錄到ssh客戶端用戶的公鑰內(nèi)容)中：cat/.ssh/id_rsa.pub/.ssh/authorized_keyshddoopubuntu:匚at/*ssh/td_rsa,pub-/,ssh/authortzed_keys|(6)現(xiàn)在即可免密碼登錄：sshlocalhosthadoopubuntu;$sshlocalhostWelcometoUbuntu14*94*1LTS(GNU/Linux3*13*G-32-genericx86_64)*Documen七a七ion:https:/help*ubuntu*con/469packag

7、escanb亡updated*294updatesaresecurityupdates.退出：exithadoopubuntu*5exttlogoutConn亡匚tiontolocolhostclosed,hadoopgubuntu:5|4、JAVA環(huán)境配置創(chuàng)建jvm文件夾：sudomkdir/usr/lib/jvm將下好的jdk文件解壓，并將解壓后的文件夾重命名為java-7-sun，：sudotarxzvfjdk-7u45-linux-i586.tar.gz-C/usr/lib/jvm/(由于安裝的Ubuntu系統(tǒng)為64位的，學(xué)長提供的jdk7u79為32位的，安裝之后無法使用，所以重新下

8、載了一個(gè)jdk,由于7u79版本官網(wǎng)下載速度太慢，因此下載了一個(gè)7u45版本的jdk)重命名：sudomvjdkl.7.0_45/java7sun/配置環(huán)境變量:sudogedit/.bashrc在最后添加如下四行：exportJAVA_HOME二/usr/lib/jvm/java-7-sunexportJRE_HOME=$JAVA_HOME/jreexportCLASSPATH二.：$JAVA_HOME/lib:$JRE_HOME/libexportPATH=$JAVA_HOME/bin:$PATH啟用配置：source/.bashrc配置默認(rèn)JDK：sudoupdate-alternati

9、ves-install/usr/lib/jvm/javajava/usr/lib/jvm/java-7-Sun/bin/java300sudoupdate-alternativesinstall/usr/lib/jvm/javacjavac/usr/lib/jvm/java7sun/bin/javac300hadoopubuntu:/uisr/lib/jvnSsudoupdate-alternatives-Install/usr/lib/jvnfjavajava/usr/lib/jvm/Java-s-giun/bin./java3update-alternatIves:using/usr/li

10、b/jjava-8-sun/btn/javatoprovide/usr/Ub/jYn/java(javain自動(dòng)模式hadoopubuntu:/usr/lib/Jvm$sudoupdiate-alternativesinstall/usr/lib/jvmfjiavac.javac/usr/lib/j/java-B-sun/lbtn/javacJ99_upda日ItEcriiiati廿亡S4using/usr/lib/jvh/java-Sj曷丁日utoprovide/usr/lib/jvm/javac:(javac)tn自動(dòng)模式hadoopubuntu:/usr/lib/jvm$|測試是否已完成

11、：javaversionhadoopubuntu:Sjava-versionJavaver&ton1-7”3_45JavaCTM)SERuntimeEnvironment(build17.0_45-bl8)avaHotspot(TM)serverVM(butld2.4.45-bos,mixedmode)hadoopubuni=u:$|(sudoupdate-alternativesconfigjava可查看當(dāng)前各種JDK的版本和配置)hadoopgubuntu:/usr/lib/jvmsudoupdate-ltriiatl廿e%-configjava鏈接蛆java(提供/usr/llb/jvm

12、/java)中只有一個(gè)悽選Jj:/usr/ltb/jvm/Java-Esun/bm/Java無需配畫“hadDopgubuntidi/usr/ltb/jvmS|5、安裝hadoop將hadoopl.2.1解壓至/usr/local下，并將其重命名為hadoop：sudotarxzvfhadoop-1.2.1.tar.gz-C/usr/local重命名：sudomvhadoop-1.2.1/hadoop/授權(quán)(讀寫權(quán)限)：sudochmod777hadoop/編輯hadoop-env.sh文件：sudogedit/usr/local/hadoop/conf/hadoop-env.sh,增加如下三

13、行：exportJAVA_HOME=/usr/lib/jvm/java-7-sunexportHAD00P_H0ME二/usr/local/hadoopexportPATH=$PATH:/usr/local/hadoop/bin如下圖所示：tiadoop-eniAshXacenotenodes.#TheJdVdreH.tiaxpctrtJAVA_HDMt=/usrylib/S-sunexpertJAVAA屮0LJIkh,j琳wnrHADOOP_HDM=,/U5Jy10亠l丿邸呂fftFAXfi=rPAiii;/lur/locwExtrCLASSPATHelements,optionjI.Wt?

14、*porthlADO口卜1_匚1_生臺(tái)呂P卡rH=執(zhí)行命令：sourcehadoop-env.sh輸入hadoop，出現(xiàn)如下圖所示，說明hadoop命令已經(jīng)可以使用:hadoopubuntw:/usr/local/hadciop/confSsourcehadoop-env.shhsdoopubuntu:/usr/lo匚ail/haidoop/czoriFEhadcopUsagie:hadoop-匚ofigconfdirCOMMANDwherecommandtsoneof:ndmenodleormatsecondarynanenodenanenodedatanadedfsadnInrnradrn

15、tnfsckfsbalancerotvfetchdtjobtrackerpipestasktrackerformattheDFSftlesystenrunthwDFSsecoinclarynanenoderun七h(yuǎn)e*QFSnamenoderunaDIFSdatanodsrunaDF5ddrntnclientrunaMap-ReduceadmtriclientrunaDFSfilesystemcheckingutilityrunagenericfLLesystemuserclt巴ntrunaclusterbalancingutilityapplytheofflinefstnageviewert

16、oanfstnagefetchadelegationtokenfromtheNameNoderuntheMapReducejobTra匚k啟廠noderunaPipesjobrunaMapReducetaskTra匚leernodeunJobhistoryerversasastandalonedaemon6、測試wordcount創(chuàng)建input文件夾：sudomkdir/usr/local/hadoop/input將hadoop文件夾下的README.txt文件拷貝至input中：sudocpREADME.txtinput執(zhí)行：hadoopjarhadoop-examplesT.2.1.jar

17、wordcountinputoutput.f*r15/O5/ZO6：3B1-2INFOmapreda.3otcT1_ent:mapHCJO晉reducetioo%is/osyz-i06:30;42INFOnopredJobCTi.ent:Jobcomplete;j06_1_0：01_6115!7716_0O0i115/5/246：3S：42INFmapred-obcltent;:Counters.:20*15/35/2QGI30IIZIMFOnapredobcTi.entxFi_TeoulzpuXFormatCounters15/0526：3S：42INFOmapred.JobCHent:By

18、tesNrlt:t:en=i32615/O5/2438；4ZINFQmapredJobGltent;:FileinputFormat;Counters15/35/246:8i42INFmapped.ZJobCTi-en七：BytesRead=136615/05/2436：1642INFOmapredJobCltent:FL1eSystzenCouhters15/5/2406：18：4ZINFOmapredobclteriit:IFIILE-BYTES-RIEADZQi15/35246i38i42INFmapped.JobCTi-entxFIILE_BYTES_WRITTEN=9565615/0

19、15/206：IS：42INFOmaipredJBabClleht:Map-ReduceiFramework15/Q5/Z4OS：3B142INFOmapreda.3obcT1_ent:ReduceInputgroups=1.31.15/05/2-46二38:42.INFOmapredJobCXi_en七二Mapoutputn云七erT.aT_i.zedlbytes=133615/5/243g：42INFOmapredobcltent:Combineoutputrecords=131L5/Q5/Z4SC：3BI-4ZINFOmapreda.JobcTi-entiMapInputirec:ord

20、s=3LL5/0i5yZ66：38：42INFOrnapred.JDtCXent:Reduceshumebynes=0i15/O5/Z438：4ZINFOmapred-obcltent;:Physicalmemory(bytessnapshot=015/asy2438142INFOmappeda.JobCTi-en11Reduceout:putrecords=151L5/5/2406：38：42INFOmapredJobcltent:SpilledRecords=26215/05/248：4ZINFOmapred-obcltent:MapoutputbytesZOS15/3524061i42I

21、NFOmappedJobClLi-en七二conmi-1t：Eduage(by七已呂=120348160115/O5/Z4O6：3B1-42IMFOmapred.JobcTi-ent:CPU-tmespent(nsJ=015/0572-438:42.INFOmopredJobCLi-en七二Vi_rtuaLnemoryfbytus)snopsho七=業(yè)15/5/246：3g：42INFmaipredlobcltent;:SPLIT_RAW_BTES1O4L5/O5/Z40G：3B142INFOmapreda.JobcT-LentxMapoutputrecords=L7915/0!5238：42

22、INFOmapred.Jotcxent:Combtnetnpotrecoirdi=17915/05/24018：4ZINFOmapredobcltent:Reduceinputrecords=1314.查看結(jié)果：catoutput/*hadoopubuntu:/usr/local/hadoop$cat：output/*(bis),H(ECCNJ1(TSJ)-1(see15DOOZ亠匸亠丁，174013)auh七tzp:/wwweWassenaar,org/1Adnlnls七atIon1ApacheiBEFORE1BISIBureau1Commerce=1Commcid1_七ControlICo

23、re1已p已廠七merit1ENCIExcep七ion1Export2For1FoundlattoniGovernmen七1Hadoop1Hadoop鼻1Industryi31已七七*v1說明單機(jī)版的hadoop已經(jīng)配置成功。7、偽分布式配置在hadoop文件夾下創(chuàng)建文件夾：tmp，hdfs,hdfs/name,hdfs/data修改data的權(quán)限：sudochmod755hdfs/data修改conf/中core-site.xml文件cpre-tm!h斗?-ityIcKihed:type-tyxIhrc-f1confIquf、上Lon.xz.I111vhfLme：rrt;/l口dLjhQE:

24、爭口日音ZaLu書-lya*propurt曹aPQpxtinprdlr/riain-=wii/Uir/local/hJdesp/tnp-=/1=-./prepert-c/conV1LguriErton*修改conf/中hdfs-site.xml文件iMirv-iUe.xmlx卜孑verctori!i.WJmrX-stylcshc-cttyficr-E*?41/Mil，shrcTcciffifItauriiLl口PutaLte-Epeetfl上propertyewerrldei-InthLsfIle_-*cizonF*Liy廠QdtIipro|pf-rty叫n硏齊口-rti?pL1k石因口門叫/riihn.*：*-11U4*-K1-/V*11IJH.-.prtyp-=i白E町TsFlUL#ri-aiPn*?aw-rhdri/n-an5r/v-fflIwv/pr口p*t=fvjinic-dFsad,t-acd1lr/nan?=Vdlu/uar/loc曰1/hqcicsapfhd個(gè)丘#p!IT*t十A./conrIguriitLon(5)修改mapred-site.xml文件retf-iktemlx?九的1vc?rLari