9、分布式文件系統(tǒng)(HDFS應(yīng)用)-學(xué)習(xí)課件

Oracle·WDPe·WDP分布式文件系統(tǒng)（HDFS應(yīng)用1）Oracle高校大數(shù)據(jù)課程系列本課目標(biāo)理解HDFS框架掌握HDFS命令行操作掌握HDFSAPI操作Coursecatalogue課程目錄HDFS框架1HDFS命令行操作2HDFSAPI操作3HDFS框架HDFS也是按照Master和Slave的結(jié)構(gòu)。分NameNode、SecondaryNameNode、DataNode這幾個角色。HDFS框架：HDFS框架NameNode：是Master節(jié)點，是大領(lǐng)導(dǎo)。管理數(shù)據(jù)塊映射；處理客戶端的讀寫請求；配置副本策略；管理HDFS的名稱空間；namenode內(nèi)存中存儲的是=fsimage+edits。SecondaryNameNode：是一個小弟，分擔(dān)大哥namenode的工作量；是NameNode的冷備份；合并fsimage和fsedits然后再發(fā)給namenode。DataNode：Slave節(jié)點，奴隸，干活的。負(fù)責(zé)存儲client發(fā)來的數(shù)據(jù)塊block；執(zhí)行數(shù)據(jù)塊的讀寫操作。fsimage:元數(shù)據(jù)鏡像文件（文件系統(tǒng)的目錄樹）edits：元數(shù)據(jù)的操作日志（針對文件系統(tǒng)做的修改操作記錄）HDFS框架：Coursecatalogue課程目錄HDFS框架1HDFS命令行操作2HDFSAPI操作3HDFSSHELL

使用這些命令，首先學(xué)會help的使用是很明智的選擇。它的應(yīng)用語法在一個已經(jīng)啟動Hadoop的HDFS服務(wù)與YARN服務(wù)的集群中，用Hadoop關(guān)鍵字加上“-help”即可列出所有HadoopShell支持的命令。HDFSSHELL：-help[user@master~]$hadoop-helpUsage:hadoop[--configconfdir]COMMANDwhereCOMMANDisoneof:fsrunagenericfilesystemuserclientversionprinttheversionjar<jar>runajarfilechecknative[-a|-h]checknativehadoopandcompressionlibrariesavailabilitydistcp<srcurl><desturl>copyfileordirectoriesrecursivelyarchive-archiveNameNAME-p<parentpath><src>*<dest>createahadooparchiveclasspathprintstheclasspathneededtogetthecredentialinteractwithcredentialprovidersHadoopjarandtherequiredlibrariesdaemonlogget/settheloglevelforeachdaemontraceviewandmodifyHadooptracingsettingsorCLASSNAMEruntheclassnamedCLASSNAMEHDFSSHELL通過hadoop–help查詢列出所有fs支持的語法規(guī)則HDFSSHELL：-help[user@master~]$hadoopfs-helpUsage:hadoopfs[genericoptions][-appendToFile<localsrc>...<dst>][-cat[-ignoreCrc]<src>...][-checksum<src>...][-chgrp[-R]GROUPPATH...][-chmod[-R]<MODE[,MODE]...|OCTALMODE>PATH...][-chown[-R][OWNER][:[GROUP]]PATH...][-copyFromLocal[-f][-p][-l]<localsrc>...<dst>][-copyToLocal[-p][-ignoreCrc][-crc]<src>...<localdst>][-count[-q][-h]<path>...][-cp[-f][-p|-p[topax]]<src>...<dst>][-createSnapshot<snapshotDir>[<snapshotName>]][-deleteSnapshot<snapshotDir><snapshotName>][-df[-h][<path>...]][-du[-s][-h]<path>...][-expunge]…….HDFSSHELL通過hadoop–help查詢列出所有fs支持的語法規(guī)則HDFSSHELL：-help[user@master~]$hadoopfs-helpUsage:hadoopfs[genericoptions][-appendToFile<localsrc>...<dst>][-cat[-ignoreCrc]<src>...][-checksum<src>...][-chgrp[-R]GROUPPATH...][-chmod[-R]<MODE[,MODE]...|OCTALMODE>PATH...][-chown[-R][OWNER][:[GROUP]]PATH...][-copyFromLocal[-f][-p][-l]<localsrc>...<dst>][-copyToLocal[-p][-ignoreCrc][-crc]<src>...<localdst>][-count[-q][-h]<path>...][-cp[-f][-p|-p[topax]]<src>...<dst>][-createSnapshot<snapshotDir>[<snapshotName>]][-deleteSnapshot<snapshotDir><snapshotName>][-df[-h][<path>...]][-du[-s][-h]<path>...][-expunge]…….lshadoopfs–ls/列出hdfs文件系統(tǒng)根目錄下的目錄和文件hadoopfs–ls–R/列出hdfs文件系統(tǒng)所有的目錄和文件puthadoopfs-put<localfile><hdfsfile>從本地文件系統(tǒng)中復(fù)制單個或多個源路徑到目標(biāo)文件系統(tǒng)。也支持從標(biāo)準(zhǔn)輸入中讀取輸入寫入目標(biāo)文件系統(tǒng)。hadoopfs–ls/experiment/datashadoopfs–put/home/admin/experiment/datas/test.txt/experiment/datashadoopfs–ls/experiment/datasgethadoopfs–get<hdfsfile><localfileordir>復(fù)制文件到本地文件系統(tǒng)。ls/home/admin/experiment/datashadoopfs–get/experiment/datas/test.txt/home/admin/experiment/datasls/home/admin/experiment/datasrmhadoopfs-rm<hdfsfile>刪除指定的文件。只刪除非空目錄和文件。hadoopfs–ls/experiment/datashadoopfs–rm/experiment/datas/test.txtrmhadoopfs–ls/experiment/datasmkdirhadoopfs-mkdir<hdfspath>只能一級一級的建目錄，父目錄不存在的話使用這個命令會報錯hadoopfs–ls/hadoopfs–mkdir/hdfsmkdirhadoopfs–ls/mkdirhadoopfs-mkdir–p<hdfspath>所創(chuàng)建的目錄如果父目錄不存在就創(chuàng)建該父目錄hadoopfs–ls/hadoopfs–mkdir–p/hdfsmkdirhadoopfs–ls/hadoopfs–ls/hdfscphadoopfs-cp<hdfsfile><hdfsfile>將文件從源路徑復(fù)制到目標(biāo)路徑。這個命令允許有多個源路徑，此時目標(biāo)路徑必須是一個目錄。hadoopfs–ls/hdfs/hdfs_shellhadoopfs–cp/experiment/datas/test.txt/hdfs/hdfs_shell/test.txtcphadoopfs–ls/hdfs/hdfs_shellmvhadoopfs-mv<hdfsfile><hdfsfile>目標(biāo)文件不能存在，否則命令不能執(zhí)行，相當(dāng)于給文件重命名并保存，源文件不存在hadoopfs–ls/experiment/datashadoopfs–mv/experiment/datas/test.txt/hdfs/hdfs_shellmvhadoopfs–ls/experiment/datashadoopfs–ls/hdfs/hdfs_shellfsckhadoopfsck<path>[-move|-delete|-openforwrite][-files[-blocks[-locations|-racks]]]運行HDFS文件系統(tǒng)檢查工具h(yuǎn)adoopfsck/experiment/datas-filesbalancerhadoopbalancer[-threshold<threshold>]運行集群平衡工具。管理員可以簡單的按Ctrl-C來停止平衡過程。hadoopbalancerdfsadminhadoopdfsadmin[-report][-safemodeenter|leave|get|wait][-refreshNodes][-finalizeUpgrade][-upgradeProgressstatus|details|force][-metasavefilename][-setQuota<quota><dirname>...<dirname>][-clrQuota<dirname>...<dirname>][-help[cmd]]運行一個HDFS的dfsadmin客戶端。hadoopdfsadmin-reportCoursecatalogue課程目錄HDFS框架1HDFS命令行操作2HDFSAPI操作3HDFSAPI創(chuàng)建目錄：獲得FileSystem對象，根據(jù)提供的API進(jìn)行相應(yīng)操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***創(chuàng)建文件夾**@paramfolder文件夾名*/publicstaticvoidmkdirs(Stringfolder)throwsIOException{Pathpath=newPath(folder);FileSystemfs=FileSystem.get(URI.create(HDFS),conf);if(!fs.exists(path)){fs.mkdirs(path);System.out.println("Create:"+folder);}fs.close();}HDFSAPI刪除目錄：獲得FileSystem對象，根據(jù)提供的API進(jìn)行相應(yīng)操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***刪除文件夾**@paramfolder文件夾名*/publicstaticvoidrmr(Stringfolder)throwsIOException{Pathpath=newPath(folder);FileSystemfs=FileSystem.get(URI.create(HDFS),conf);fs.deleteOnExit(path);System.out.println("Delete:"+folder);fs.close();}HDFSAPI重命名文件：獲得FileSystem對象，根據(jù)提供的API進(jìn)行相應(yīng)操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***重命名文件*@paramsrc源文件名*@paramdst目標(biāo)文件名**/publicstaticvoidrename(Stringsrc,Stringdst)throwsIOException{Pathname1=newPath(src);Pathname2=newPath(dst);FileSystemfs=FileSystem.get(URI.create(HDFS),conf);fs.rename(name1,name2);System.out.println("Rename:from"+src+"to"+dst);fs.close();}HDFSAPI列出目錄下的文件信息：獲得FileSystem對象，根據(jù)提供的API進(jìn)行相應(yīng)操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***列出該路徑的文件信息*@paramfolder文件夾名*/publicstaticvoidls(Stringfolder)throwsIOException{Pathpath=newPath(folder);FileSystemfs=FileSystem.get(URI.create(HDFS),conf);FileStatus[]list=fs.listStatus(path);System.out.println("ls:"+folder);System.out.println("=======================");for(FileStatusf:list){System.out.printf("name:%s,folder:%s,size:%d\n",f.getPath(),f.isDirectory(),f.getLen());}System.out.println("===============");fs.close();}HDFSAPI創(chuàng)建文件：獲得FileSystem對象，根據(jù)提供的API進(jìn)行相應(yīng)操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();publicstaticvoidcreateFile(Stringfile,Stringcontent)throwsIOException{FileSystemfs=FileSystem.get(URI.create(HDFS),conf);byte[]buff=content.getBytes();FSDataOutputStreamos=null;try{os=fs.create(newPath(file));os.write(buff,0,buff.length);System.out.println("Create:"+file);}finally{if(os!=null)os.close();}fs.close();}HDFSAPI上傳本地文件到HDFS：獲得FileSystem對象，根據(jù)提供的API進(jìn)行相應(yīng)操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***上傳本地文件到hdfs**@paramlocal本地文件路徑*@paramremotehdfs目標(biāo)路徑*/publicstaticvoidcopyFile(Stringlocal,Stringremote)throwsIOException{FileSystemfs=FileSystem.get(URI.create(HDFS),conf);fs.copyFromLocalFile(newPath(local),newPath(remote));System.out.println("copyfrom:"+local+"to"+remote);fs.close();}HDFSAPI下載HDFS文件到本地：獲得FileSystem對象，根據(jù)提供的API進(jìn)行相應(yīng)操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***從hdfs下載文件到本地**@paramremotehdfs文件路徑*@paramlocal本地目標(biāo)路徑*/publicstaticvoiddownload(Stringremote,