MapReduce編程環(huán)境設(shè)置和注意事項(xiàng)2010說課材料

1、Good is good, but better carries it.精益求精，善益求善。MapReduce編程環(huán)境設(shè)置和注意事項(xiàng)2010-MapReduce編程環(huán)境設(shè)置和注意事項(xiàng)遠(yuǎn)程登陸服務(wù)器請同學(xué)們使用服務(wù)器分配的原始密碼通過SSH工具登陸到實(shí)驗(yàn)集群bl6000的登陸節(jié)點(diǎn)：9，然后使用命令passwd對初始密碼進(jìn)行修改。PS：Linux下密碼不可見。接著在自己的賬號下使用命令mkdir分別建立1個(gè)目錄：MapReduce用來存放不同實(shí)驗(yàn)數(shù)據(jù)和代碼。設(shè)置編譯環(huán)境使用命令vi/.bash_profile設(shè)置環(huán)境變量，添加路徑如下文，然后使用./.bash_profile（注意前有個(gè)點(diǎn)+空格）

2、使設(shè)置生效，整個(gè)操作過程如下:ming.taobl6000$cat.bash_profile#.bash_profile#Getthealiasesandfunctionsif-f/.bashrc;then./.bashrcfi#UserspecificenvironmentandstartupprogramsPATH=/opt/gcc-4.2.4/bin:/opt/mpich-1.2.7pl/bin:/opt/intel/Compiler/11.0/069/bin/ia32:$PATH:$HOME/bin:/usr/sbinLD_LIBRARY_PATH=/opt/gcc-4.2.4/lib

3、64:/usr/lib:opt/mpich-1.2.7pl/lib/:/opt/intel/Compiler/11.0/069/lib/ia32:$LD_LIBRARY_PATHHADOOP_CLASSPATH=/opt/hadoop/hadoop-0.20.1/hadoop-0.20.1-core.jar:/opt/hadoop/hadoop/hadoop-0.20.1/lib/:/opt/hadoop/hadoop-0.20.1/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop-0.20.1/lib/jetty-6.1.14.jar:/opt/hadoo

4、p/hadoop-0.20.1/lib/jetty-util-6.1.14.jarexportPATHexportLD_LIBRARY_PATHexportHADOOP_CLASSPATHunsetUSERNAMEming.taobl6000$ming.taobl6000$./.bash_profile然后使用echo$HADOOP_CLASSPATH命令檢測環(huán)境變量是否設(shè)置成功,正常的顯示如下：ming.taobl6000$echo$HADOOP_CLASSPATH/opt/hadoop/hadoop-0.20.1/hadoop-0.20.1-core.jar:/opt/hadoop/had

5、oop/hadoop-0.20.1/lib/:/opt/hadoop/hadoop-0.20.1/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop-0.20.1/lib/jetty-6.1.14.jar:/opt/hadoop/hadoop-0.20.1/lib/jetty-util-6.1.14.jar跳轉(zhuǎn)計(jì)算節(jié)點(diǎn)本次實(shí)驗(yàn)除了bl6000的登陸頭節(jié)點(diǎn)（9）還有4個(gè)計(jì)算節(jié)點(diǎn)，分別是：c0-0,c0-1,c0-2,c0-3使用命令ssh可以從當(dāng)前節(jié)點(diǎn)轉(zhuǎn)到計(jì)算節(jié)點(diǎn),例如sshc0-1則可從當(dāng)前節(jié)點(diǎn)轉(zhuǎn)到計(jì)算節(jié)點(diǎn)c0-1,其他如此類推。編譯運(yùn)行MapReduce程序

6、示例如何編譯運(yùn)行MapReduce程序以附錄的程序WordCount.java為例:ming.taobl6000MapReduce$lsWordCount.javaming.taobl6000MapReduce$javac-classpath$HADOOP_CLASSPATHWordCount.javaming.taobl6000MapReduce$lsWordCount.classWordCount$IntSumReducer.classWordCount.javaWordCount$TokenizerMapper.classming.taobl6000MapReduce$jarcvfwd.

7、jar*.classming.taobl6000MapReduce$viinputdataming.taobl6000MapReduce$catinputdataming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-putinputdataInput-dataming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-lsFound8items-rw-r-r-3ming.taosupergroup462009-12-2818:39/user/ming.t

8、ao/Input-dataming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-cat/user/ming.tao/Input-dataThisisaMapReduceExp.HadoopMapRedueHDFSming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopjarwd.jarWordCountInput-dataOutput-dir09/12/2818:46:12INFOinput.FileInputFormat:Totalinputpathsto

9、process:109/12/2818:46:13INFOmapred.JobClient:Runningjob:job_200912242113_000509/12/2818:46:14INFOmapred.JobClient:map0%reduce0%ming.taobl6000MapReduce$ming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-lsFound10items-rw-r-r-3ming.taosupergroup462009-12-2818:39/user/ming.tao/Input-dirdr

10、wxrwxrwx-ming.taosupergroup02009-12-2818:46/user/ming.tao/Output-dirming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-ls/user/ming.tao/Output-dirFound2itemsdrwxrwxrwx-ming.taosupergroup02009-12-2818:46/user/ming.tao/Output-dir/_logs-rw-r-r-3ming.taosupergroup622009-12-2818:46/user/ming

11、.tao/Output-dir/part-r-00000ming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-cat/user/ming.tao/Output-dir/part-r-00000Exp.1HDFS1Hadoop1MapReduce2This1a1is1ming.taobl6000MapReduce$五.編譯運(yùn)行MapReduce程序5.1編譯：javac-classpath$HADOOP_CLASSPATyourprog.java我們采用的Java版本是：javaversion1.6.0_145.2打包：j

12、arcvf*.jar*.class5.3運(yùn)行：/opt/hadoop/hadoop-0.20.1/bin/hadoopjar*.jaryourprogInput-dataOutput-dir注意：如果你需要用bin/hadoopjar命令，需要在源程序中加入：Job.setJarByClass(yourprog.class);可參見demoprogram下的WordCount.java文件(job.setJarByClass(WordCount.class);)一些常見問題如何將本地文件放入HDFS文件系統(tǒng)中對策：將本地的localfile文件拷貝到HDFS文件系統(tǒng)的HDFSfile運(yùn)用/op

13、t/hadoop/hadoop-0.20.1/bin/hadoopdfsputlocalfileHDFSfile或/opt/hadoop/hadoop-0.20.1/bin/hadoopdfscopyFromLocallocalfileHDFSfile6.2如何將本地文件放入HDFS文件系統(tǒng)中對策：將HDFS文件系統(tǒng)的HDFSfile拷貝到本地的localfile文件運(yùn)用/opt/hadoop/hadoop-0.20.1/bin/hadoopdfsgetHDFSfilelocalfile或/opt/hadoop/hadoop-0.20.1/bin/hadoopdfscopyToLocalHDFSfilelocalfile6.3如何將在HDFS文件系統(tǒng)中創(chuàng)建目錄對策：將HDFS文件系統(tǒng)創(chuàng)建mydir目錄運(yùn)用/opt/hadoop/hadoop-0.20.1/bin/hadoopdfsmkdirmydir6.4如何查看HDFS中的文件及文件內(nèi)容對策：運(yùn)用/opt/hadoop/hadoop-0.20.1/bin/hadoopdfsls/user/you_account