大數(shù)據(jù)綜合實(shí)訓(xùn)實(shí)驗(yàn)手冊(cè)-項(xiàng)目實(shí)施

高職大數(shù)據(jù)綜合實(shí)訓(xùn)實(shí)驗(yàn)手冊(cè)目錄 2.1實(shí)驗(yàn)?zāi)康?2.2實(shí)驗(yàn)要求 2.3實(shí)驗(yàn)環(huán)境 2.4試驗(yàn)過程 3.1實(shí)驗(yàn)?zāi)康?203.2實(shí)驗(yàn)要求 213.3實(shí)驗(yàn)環(huán)境 3.4試驗(yàn)過程 21 步驟二：安裝依賴的組件 22 22 22步驟六：起停數(shù)據(jù)庫(kù) 步驟八：設(shè)置允許遠(yuǎn)程連接 3.4.2實(shí)驗(yàn)任務(wù)二：簡(jiǎn)單操作MySQL 步驟一：連接數(shù)據(jù)庫(kù) 步驟二：查看當(dāng)前登錄用戶 步驟三：查看數(shù)據(jù)庫(kù)清單 步驟四：創(chuàng)建數(shù)據(jù)庫(kù) 步驟六：創(chuàng)建一張表 步驟七：查看數(shù)據(jù)庫(kù)的表清單 步驟八：查看表結(jié)構(gòu) 步驟九：插入一條數(shù)據(jù) 0步驟十：查詢表數(shù)據(jù) 3.4.3實(shí)驗(yàn)任務(wù)二：使用MySQL常用工具集 3.4.4實(shí)驗(yàn)任務(wù)四：備份與恢復(fù)MySQL數(shù)據(jù)庫(kù) 步驟二：重建數(shù)據(jù)庫(kù) 步驟三：從備份文件恢復(fù)數(shù)據(jù)庫(kù) 3.4.5實(shí)驗(yàn)任務(wù)五：安裝和配置Redis 步驟一：安裝gcc 38 393.4.6實(shí)驗(yàn)任務(wù)六：使用Redis 3.4.7實(shí)驗(yàn)任務(wù)七：維護(hù)Redis 41 413.4.8實(shí)驗(yàn)任務(wù)八：安裝和配置zookeeperl偽集群模式) 45 45步驟二：解壓安裝三個(gè)節(jié)點(diǎn) 46步驟三：修改zookeeper1節(jié)點(diǎn)的配置文件 46 46步驟三：修改zookeeper2節(jié)點(diǎn)的配置文件 47 47步驟三：修改zookeeper3節(jié)點(diǎn)的配置文件 48 48 480步驟五：驗(yàn)證安裝結(jié)果 3.4.9實(shí)驗(yàn)任務(wù)九：安裝和配置kafka 49 49 49步驟三：修改節(jié)點(diǎn)1的配置 步驟四：新增并修改節(jié)點(diǎn)2的配置 步驟五：新增并修改節(jié)點(diǎn)3的配置 步驟七：驗(yàn)證安裝結(jié)果 步驟八：驗(yàn)證發(fā)送消息 步驟八：驗(yàn)證接收消息 4實(shí)驗(yàn)三：大數(shù)據(jù)集群環(huán)境安裝、配置與基本管理 錯(cuò)誤!未定義書簽。4.1實(shí)驗(yàn)?zāi)康?錯(cuò)誤!未定義書簽。4.2實(shí)驗(yàn)要求 錯(cuò)誤!未定義書簽。4.3實(shí)驗(yàn)環(huán)境 錯(cuò)誤!未定義書簽。4.4試驗(yàn)過程………………錯(cuò)誤!未定義書簽。4.4.1實(shí)驗(yàn)任務(wù)一：部署前準(zhǔn)備……錯(cuò)誤未定義書簽。步驟一：禁用防火墻………………錯(cuò)誤!未定義書簽。步驟三：安裝輔助工具……………錯(cuò)誤!未定義書簽。4.4.2實(shí)驗(yàn)任務(wù)二：安裝和配置Hadoop…………錯(cuò)誤未定義書簽。步驟二：配置環(huán)境變量……………錯(cuò)誤!未定義書簽。步驟三：激活環(huán)境變量…………步驟四：準(zhǔn)備數(shù)據(jù)目錄……………錯(cuò)誤!未定義書簽。4.4.3實(shí)驗(yàn)任務(wù)三：安裝和配置Hive………………錯(cuò)誤未定義書簽。4.4.4實(shí)驗(yàn)任務(wù)四：安裝和配置Scala………………錯(cuò)誤未定義書簽。步驟三：配置環(huán)境變量…………步驟四：激活環(huán)境變量…………步驟五：驗(yàn)證安裝結(jié)果……………錯(cuò)誤|未定義書簽。4.4.5實(shí)驗(yàn)任務(wù)五：安裝和配置Spark………………錯(cuò)誤未定義書簽。 5實(shí)驗(yàn)四：基于網(wǎng)頁(yè)爬蟲的數(shù)據(jù)采集、處理和數(shù)據(jù)存儲(chǔ) 725.1實(shí)驗(yàn)?zāi)康?5.2實(shí)驗(yàn)要求 5.3實(shí)驗(yàn)環(huán)境 5.4試驗(yàn)過程 5.4.1實(shí)驗(yàn)任務(wù)一：安裝Scrapy運(yùn)行環(huán)境 步驟二：安裝pip包管理工具 步驟四：安裝輔助工具 5.4.2實(shí)驗(yàn)任務(wù)二：網(wǎng)站結(jié)構(gòu)分析及爬蟲算法設(shè)計(jì) 步驟二：分析網(wǎng)站結(jié)構(gòu) 步驟三：設(shè)計(jì)資源抓取算法 5.4.3實(shí)驗(yàn)任務(wù)三：搭建爬蟲項(xiàng)目 步驟一：創(chuàng)建項(xiàng)目 78步驟二：新建爬蟲程序 80步驟四：配置爬蟲 步驟五：運(yùn)行爬蟲 5.4.4實(shí)驗(yàn)任務(wù)四：將爬取的數(shù)據(jù)存儲(chǔ)到MySQL 步驟二：修改爬蟲配置 步驟三：重新運(yùn)行爬蟲 5.4.5實(shí)驗(yàn)任務(wù)五：將爬取的數(shù)據(jù)存儲(chǔ)到文件 步驟一：修改爬蟲配置 5.4.6實(shí)驗(yàn)任務(wù)六：安裝flume-ng 步驟三：配置環(huán)境變量 步驟四：激活環(huán)境變量 步驟五：驗(yàn)證安裝結(jié)果 5.4.7實(shí)驗(yàn)任務(wù)六：通過flume-ng將文件數(shù)據(jù)導(dǎo)入到kofka 步驟一：創(chuàng)建數(shù)據(jù)源目錄 步驟二：準(zhǔn)備數(shù)據(jù)文件 步驟四：執(zhí)行任務(wù) 步驟五：驗(yàn)證執(zhí)行 5.4.8實(shí)驗(yàn)任務(wù)七：通過flume-ng將kafka數(shù)據(jù)導(dǎo)出到文件 步驟五：驗(yàn)證執(zhí)行 6.1實(shí)驗(yàn)?zāi)康?6.2實(shí)驗(yàn)要求 6.3實(shí)驗(yàn)環(huán)境 6.4實(shí)驗(yàn)過程 6.4.1試驗(yàn)任務(wù)一：安裝Kettle 6.4.2實(shí)驗(yàn)任務(wù)二：新建從MySQl到文件的轉(zhuǎn)換 步驟三：新建輸入 步驟四：新建輸出 步驟五：測(cè)試轉(zhuǎn)換 6.4.3實(shí)驗(yàn)任務(wù)三：新建和配置作業(yè) 步驟一：新建作業(yè) 6.4.4實(shí)驗(yàn)任務(wù)四：安裝和配置Sqoop 步驟四：驗(yàn)證安裝結(jié)果 步驟六：驗(yàn)證修改配置的結(jié)果 6.4.5實(shí)驗(yàn)任務(wù)五：將MySQL數(shù)據(jù)導(dǎo)入到HDFS 步驟一：檢查HDFS中目錄是否存在 步驟三：查看導(dǎo)入結(jié)果 6.4.6實(shí)驗(yàn)任務(wù)六：將HDFS數(shù)據(jù)導(dǎo)出到MySQL 109步驟三：查看導(dǎo)出結(jié)果 7實(shí)驗(yàn)六：數(shù)據(jù)清洗 7.1實(shí)驗(yàn)?zāi)康?7.2實(shí)驗(yàn)要求 7.3實(shí)驗(yàn)環(huán)境 7.4實(shí)驗(yàn)過程 7.4.1實(shí)驗(yàn)任務(wù)一：Spark實(shí)現(xiàn)數(shù)據(jù)清洗 步驟四：Spark處理字段非法值 步驟五：Spark移除重復(fù)數(shù)據(jù) 112 7.4.2實(shí)驗(yàn)任務(wù)三：發(fā)布spark數(shù)據(jù)清洗服務(wù) 步驟一：本地打包 步驟二：上傳至服務(wù)器 步驟三：運(yùn)行服務(wù) 7.4.3實(shí)驗(yàn)任務(wù)二：MapReduce實(shí)現(xiàn)離線分析 步驟二：搭建項(xiàng)目框架 步驟六：分組完整代碼實(shí)現(xiàn) 7.4.4實(shí)驗(yàn)任務(wù)四：發(fā)布MapReduce離線分析服務(wù) 步驟一：本地打包 步驟二：上傳至服務(wù)器 步驟三：運(yùn)行服務(wù) 8實(shí)驗(yàn)七：數(shù)據(jù)倉(cāng)庫(kù)的建立 8.1實(shí)驗(yàn)?zāi)康?8.2實(shí)驗(yàn)要求 8.3實(shí)驗(yàn)環(huán)境 8.4實(shí)驗(yàn)過程 8.4.1實(shí)驗(yàn)任務(wù)一：檢查Hive運(yùn)行環(huán)境 步驟三：檢查數(shù)據(jù)是否準(zhǔn)備好 8.4.2試驗(yàn)任務(wù)二：設(shè)計(jì)數(shù)據(jù)結(jié)構(gòu) 步驟二：根據(jù)返回結(jié)果確定數(shù)據(jù)結(jié)構(gòu) 8.4.3實(shí)驗(yàn)任務(wù)三：創(chuàng)建庫(kù)和表 步驟一：進(jìn)入Hive環(huán)境 步驟二：創(chuàng)建數(shù)據(jù)庫(kù) 步驟三：創(chuàng)建表 8.4.4實(shí)驗(yàn)任務(wù)二：數(shù)據(jù)入庫(kù) 步驟二：裝載數(shù)據(jù)入庫(kù) 9實(shí)驗(yàn)八：數(shù)據(jù)挖掘 9.1實(shí)驗(yàn)?zāi)康?9.2實(shí)驗(yàn)要求 9.3實(shí)驗(yàn)環(huán)境 9.4實(shí)驗(yàn)過程 9.4.1實(shí)驗(yàn)任務(wù)一：檢查Hive及表數(shù)據(jù) 步驟一：登錄數(shù)據(jù)庫(kù) 步驟二：檢查表數(shù)據(jù) 9.4.2實(shí)驗(yàn)任務(wù)二：建立匯聚視圖 步驟一：按月統(tǒng)計(jì)教材 步驟二：驗(yàn)證統(tǒng)計(jì)結(jié)果 9.4.3實(shí)驗(yàn)任務(wù)三：建立分析視圖 步驟一：分析最受歡迎的10本教材 132步驟二：驗(yàn)證分析結(jié)果 9.4.4實(shí)驗(yàn)任務(wù)四：將分析結(jié)果導(dǎo)出到MySQl 133 步驟三：驗(yàn)證導(dǎo)出結(jié)果 10實(shí)驗(yàn)九：數(shù)據(jù)可視化 10.1實(shí)驗(yàn)?zāi)康?10.2實(shí)驗(yàn)要求 10.3實(shí)驗(yàn)環(huán)境 10.4實(shí)驗(yàn)過程 10.4.1實(shí)驗(yàn)任務(wù)一：準(zhǔn)備基礎(chǔ)開發(fā)環(huán)境、組件和數(shù)據(jù) 步驟四：檢查可視化數(shù)據(jù)源的數(shù)據(jù)情況 10.4.2實(shí)驗(yàn)任務(wù)二：開發(fā)WEB可視化項(xiàng)目 141 10.4.3實(shí)驗(yàn)任務(wù)三：本地運(yùn)行項(xiàng)目 11實(shí)驗(yàn)十：大數(shù)據(jù)運(yùn)維 11.1.1實(shí)驗(yàn)任務(wù)一：備份NameNode的元數(shù)據(jù) 錯(cuò)誤!未定義書簽。步驟一：設(shè)置冗余地址 11.1.2實(shí)驗(yàn)任務(wù)二：DataNode塊掃描器 錯(cuò)誤!未定義書簽。 11.1.3實(shí)驗(yàn)任務(wù)三：負(fù)載器均衡調(diào)整………………錯(cuò)誤!未定義書簽。11.1.4實(shí)驗(yàn)任務(wù)四：例行停止機(jī)器……錯(cuò)誤未定義書簽。11.1.5實(shí)驗(yàn)任務(wù)五：集群添加服務(wù)器及服務(wù)節(jié)點(diǎn)…錯(cuò)誤未定義書簽。11.1.6實(shí)驗(yàn)任務(wù)六：集群節(jié)點(diǎn)性能測(cè)試……………錯(cuò)誤未定義書簽。11.1.7實(shí)驗(yàn)任務(wù)七：平臺(tái)日志查看及審計(jì)…………錯(cuò)誤未定義書簽。11.1.8實(shí)驗(yàn)任務(wù)八：設(shè)置日志級(jí)別……錯(cuò)誤未定義書簽。11.1.9實(shí)驗(yàn)任務(wù)九：清理臨時(shí)日志……錯(cuò)誤未定義書簽。11.1.10實(shí)驗(yàn)任務(wù)十：定期執(zhí)行數(shù)據(jù)均衡腳本………錯(cuò)誤未定義書簽。11.1.11實(shí)驗(yàn)任務(wù)十一：Namenode的高可靠性……錯(cuò)誤!未定義書簽。11.1.12實(shí)驗(yàn)任務(wù)十二：HDFSHA………錯(cuò)誤未定義書簽。11.1.13實(shí)驗(yàn)任務(wù)十三：大數(shù)據(jù)管理平臺(tái)HA…………錯(cuò)誤未定義書簽。11.1.14實(shí)驗(yàn)任務(wù)十五：Ganglia監(jiān)控軟件……………錯(cuò)誤!未定義書簽。步驟五：界面訪問………………錯(cuò)誤!未定義書簽。11.1.15實(shí)驗(yàn)任務(wù)十六：操作系統(tǒng)調(diào)優(yōu)………………錯(cuò)誤未定義書簽。11.1.16實(shí)驗(yàn)任務(wù)十七：JVM調(diào)優(yōu)……錯(cuò)誤未定義書簽。11.1.17實(shí)驗(yàn)任務(wù)十八：HDFS調(diào)優(yōu)……錯(cuò)誤!未定義書簽。11.1.18實(shí)驗(yàn)任務(wù)十九：MapReduce調(diào)優(yōu)……………錯(cuò)誤未定義書簽。11.1.19實(shí)驗(yàn)任務(wù)二十：Yarn調(diào)優(yōu)……錯(cuò)誤未定義書簽。11.1.20實(shí)驗(yàn)任務(wù)二十一：Hive性能調(diào)優(yōu)……………錯(cuò)誤!未定義書簽。11.1.21實(shí)驗(yàn)任務(wù)二十二：HBASE調(diào)優(yōu)………………錯(cuò)誤未定義書簽。項(xiàng)目經(jīng)理系統(tǒng)架構(gòu)師測(cè)試工程師運(yùn)維工程師●熟悉常用Linux操作系統(tǒng)命令●熟悉JDK的安裝和環(huán)境變量配置本實(shí)驗(yàn)所需資源環(huán)境如表1-1所示。表1-1資源環(huán)境最低配置：雙核CPU、8GB內(nèi)存、100G硬盤CentOS-7-x86_64-Everything-1810CamtOS7lsta11Cents2imstthisRdisaimstalltentas7Trubleshootis選擇語(yǔ)言：EnglishEnglish(UnitedStates)English(UnitedKingdom)English(Iindia)English(Australia)English(Canada)English(Denmark)English(lireland)English(NewZealand)English(Nigeria)English(HongKongSARChina)rinLaCENTOS7INSTALLATIONWELCOMETOCENTOS7.Whatlanguagewouldyouliketouseduringtheinstallationprocess?EnglehEnglishAfrikaansAfrikaansA-c管AmhancArabic西AssamestAsturianuAsturlan6enapycxanBelaruslarGenrapcxmBulgarian對(duì)RBengal@OutCenn設(shè)置時(shí)間：77INSTALLATIONHeplCentOSUSERSETTINGSROOTPASSWORDRootpasswordisnotsetmmUSERCREATIONNouserwlllbecreatedCENTOSCONFIGURATIONHROOTPASSWORDCENTOS7ROOTPASSWORDHePTherootakcoustisusedforatninsteringthesystem.Enterapasswordfortherootuser,StrongConfirt…………CREATEUSERCENTOS7INSTALLATIONusHelpTip:Keepyourusernameshorterthan32dwaxtersanddonotusespexes.?MakethisuseradministratorPassword……WeakConfirmpassword……………Advanced.TpdvuePdtwIhePesordcenonetheunetnwtonefoauweteceU等待安裝完成：CENTOS7INSTALLATIONCENTOS7INSTALLATIONuHelpCentOSUSERSETTINGSROOTPASSWORDRootpasswordissetGosheadandreboottostastusingitRetootUSERCREATIONAdministratorgzdsjwlbecreatedCONFIGURATIONCentosLimx?(Core)locathostloyin:gzdsjPassunrd:entasLimx7(Core)Neriel3.16.8-957.617.×86.61omam×6661tlogintgrdsjhone42d?Jtgds.jlacsthost-1步驟四：配置存儲(chǔ)空間停止防火墻：[root@localhost~]#systemctlstopfirewalld禁用防火墻：步驟六：創(chuàng)建賬號(hào)及密碼，修改賬號(hào)權(quán)限通過文件上傳工具將安裝文件jdk-8u201-linux-x64.tar.gz上傳到/opt/hadoop目錄步驟二：解壓安裝JDK執(zhí)行以下命令創(chuàng)建安裝目錄：執(zhí)行以下命令解壓JDK安裝包tar-zxf/opt/hadoop/jdk-8u201-linux-x64.tar.gz執(zhí)行以下命令創(chuàng)建鏈接新增或編輯/etc/profile.d/java.sh文件，添加以下內(nèi)容：步驟四：激活環(huán)境變量執(zhí)行以下命令激活環(huán)境變量：執(zhí)行以下命令檢查java版本信息，驗(yàn)證是否安裝成功：完成本實(shí)驗(yàn)，您應(yīng)該能夠：●掌握MySQL的安裝和基本配置●掌握Redis的安裝和基本配置●熟悉常用Linux操作系統(tǒng)命令最低配置：雙核CPU、8GB內(nèi)存、100G硬盤CentOS-7-x86_64-Everything-1810組件MySQL、redis、jdk、zookeeper、kafka、gcc及其依賴包yumremovemariadb查詢已安裝的MariaDB組件：根據(jù)查詢結(jié)果依次執(zhí)行以下命令刪除查詢到的組件：或者離線安裝net-tools,將net-tools-2.0-0.24.20131004git.el7.x86_64.rpm上傳到/opt目錄，然后執(zhí)行以下命令：通過文件上傳工具將安裝文件mysql-5.7.25-1.el7.x86_64.rpm-bundle.tar上傳到執(zhí)行以下命令解壓MySQL安裝包c(diǎn)d/opt執(zhí)行以下命令安裝MySQL相關(guān)組件：servicemysqldstatus啟動(dòng)MySQL:servicemysqldstartservicemysqldstop重啟MySQL:servicemysqldrestartgreppassword/var/log/mysqld.logmysql>setpassword=password("Gzdsj_1234");mysql_upgrade-uroot-pgzdsj1234登錄MySQL,執(zhí)行以下命令：mysql>GRANTALLPRIVILEGESON**TO'root'@'%'IDENTIFIEDBY'Gzdsj_1234'WITHmysql>flushprivileges;[root@master2usr]#mysql-hlocalhost-P3306-uroot-pGzdsj_1mysql:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure.WelcometotheMySQLmonitor.Commandsendwith;or\g.Copyright(c)2000,2019,Oracleand/oritsaffliates.Allrightsreserved.OracleisaregisteredtrademarkofOracleCorporationand/oritsaffiliates.Othernamesmaybetrademarksoftheirrespectiveowners.Type'help;'orVh'forhelp.Type\c'toclearthecurrentinputstatement.步驟二：查看當(dāng)前登錄用戶mysql>selectcurrent_user();++|current_user)|++++步驟三：查看數(shù)據(jù)庫(kù)清單mysql>showdatabases;+ ++ +linformation_schemal|performance_schema|+創(chuàng)建數(shù)據(jù)庫(kù)：mysql>createdatabasegzdsj;QueryOK,1rowaffected(0.00sec)查看創(chuàng)建結(jié)果：mysql>showdatabases;++++|information_schema|+使用數(shù)據(jù)庫(kù)：mysql>usegzdsjDatabasechanged查看當(dāng)前數(shù)據(jù)庫(kù)：mysql>selectdatabase();++|database()|++++mysql>createtablet_file_info(->idvarchar(128),->namevarchar(256)->);QueryOK,0rowsaffected(0.04sec)mysql>showtables++|Tables_in_gzdsj|++++步驟八：查看表結(jié)構(gòu)mysql>desct_file_info;++++++++++++mysql>insertintot_file_info(id,name)values('abc','test_file');QueryOK,1rowaffected(0.00sec)mysql>select*fromt_file_info;+++++++++[root@localhostopt]#mysqladmin-uroot-poldpasswordpassword'new步驟二：檢查MySQLServer是否可用[root@localhostopt]#mysqladmin-uroot-pGzdsj_1234pingmysqladmin:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure.[root@localhostopt]#步驟三：查詢服務(wù)器版本[root@localhostopt]#mysqladmin-uroot-pGzdsj_1234versionmysqladmin:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure.mysqladminVer8.42Distrib5.7.26,forLinuxonx86_64Copyright(c)2000,2019,Oracleand/oritsaffiliates.Allrightsreserved.OracleisaregisteredtrademarkofOracleCorporationand/oritsaffiliates.Othernamesmaybetrademarksoftheirrespectiveowners.ConnectionUNIXsocket/var/lib/mysql/mysql.sockThreads:2Questions:2848Slowqueries:0Opens:277Flushtables:1Opentables:42Queriespersecondavg:0.239[root@localhostopt]#步驟四：檢查服務(wù)器的當(dāng)前狀態(tài)每個(gè)兩秒查看一次服務(wù)器的狀態(tài)，總共重復(fù)5次：[root@localhostopt]#mysqladmin-uroot-pGzdsj_1234-i2-c5statusEnterpassword:Uptime:10913Threads:2Questions:2826Slowqueries:0Opens:277FlushOpentables:42Queriespersecondavg:0.258 Uptime:10915Threads:2Questions:2827Opentables:42Queriespersecondavg:0.259 Uptime:10917Threads:2Questions:2828Opentables:42Queriespersecondavg:0.259 Uptime:10919Threads:2Questions:2829Opentables:42Queriespersecondavg:0.259queries:0queries:0queries:0Opens:277Opens:277Opens:277Uptime:10921Threads:2Questions:2830Slowqueries:0Opens:277FlushOpentables:42Queriespersecondavg:0.259[root@localhostopt]#tables:1tables:1tables:1tables:1tables:1步驟五：查看執(zhí)行的SQL語(yǔ)句信息[root@localhostopt]#mysqladmin-uroot-pGzdsj_1234processlistmysqladmin:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure.++++++++|Id|User|Host|db|Command++++++++++++創(chuàng)建數(shù)據(jù)庫(kù)：[root@localhostopt]#mysqladmin-uroot-pGzdsj_1234createdbname顯示服務(wù)器上所有數(shù)據(jù)庫(kù)：[root@localhostopt]#mysqmysqlshow:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure. +Databases++|information_schema||performance_schema|+[root@localhostopt]#mysqlshow-uroot-pGzdsj_1234mysqlmysqlshow:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure.Database:mysql+·Tables|columns_priv|engine_costlgeneral_log|gtid_executed|help_keyword|help_relation|innodb_index_stats|innodb_table_stats|ndb_binlog_index|proxies_priv|server_cost|slave_master_info|slave_worker_info+統(tǒng)計(jì)數(shù)據(jù)庫(kù)中各表的列數(shù)和行數(shù)[root@localhostopt]#mysqlshow-uroot-pGzdsj_1234mysql-v-vmysqlshow:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure+columns_privengine_costhelp_keywordhelp_relationinnodb_table_statsslave_worker_infoA| +步驟七：關(guān)閉MySQL服務(wù)[root@localhostopt]#mysqladmin-uroot-pGzdsj_1234shutdown步驟一：備份數(shù)據(jù)庫(kù)備份數(shù)據(jù)庫(kù)：查看備份文件：[root@master2usr]#moregzdsj.sql--Host:localhostDatabase:gzds/*!40101SET@OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION*/;/*!40101SETNAMESutf8*/;/*!40103SET@OLD_TIME_ZONE=@@TIME_ZONE*/;/*!40103SETTIME_ZONE='+00:00'*/;/*!40014SET@OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS,UNIQUE_CHECKS=0*/;FOREIGN_KEY_CHECKS=0*/;SQL_MODE='NO_AUTO_VALUE_ON_ZERO'*/;/*!40111SET@OLD_SQL_NOTES=@@SQL_NOTES,SQL_NOTES=0*/;--Tablestructurefortablet_file_info`/*!40101SETcharacter_set_client=utf8*/;CREATETABLE`t_file_info`(id'varchar(128)DEFAULTNULL,'name'varchar(256)DEFAULTNULL)ENGINE=InnoDBDEFAULTCHARSET=latin1;/*!40101SETcharacter_set_client=@saved_cs_client*/;--Dumpingdatafortabletfile_info`LOCKTABLESt_file_info'WRITE;/*!40000ALTERTABLE`t_file_info`DISABLEKEYS*/;INSERTINTO't_file_info`VALUES(abc',test_file');/*!40000ALTERTABLEt_file_info'ENABLEKEYS*/;/*!40103SETTIME_ZONE=@OLD_TIME_ZONE*/;/*!40101SETSQLMODE=@OLD_SQL_MODE*/;/*!40101SETCOLLATION_CONNECTION=@OLD_COLLATION_CONNECT--Dumpcompletedon2019-03-0311:49:12[root@master2usr]#mysql-uroot-pGzdsj1234mysql:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure.WelcometotheMySQLmonitor.Commandsendwith;or\g.YourMySQLconnectionServerversion:5.7.25MySQLCommunityServer(GPL)Copyright(c)2000,2019,Oracleand/oritsaffiliates.Allrightsreserved.OracleisaregisteredtrademarkofOracleCorporationand/oritsaffiliates.Othernamesmaybetrademarksoftheirrespectiveowners.Type'help;'or\h'forhelp.Type\c'toclearthecurrentinputstatement.mysql>dropdatabasegzdsj;QueryOK,1rowaffected(0.mysql>showdatabases;++++|information_schema||performance_schema|+mysql>createdatabasegzdsj;QueryOK,1rowaffected(0.mysql>usegzdsj;mysql>showtables;Emptyset(0.00sec)恢復(fù)數(shù)據(jù)庫(kù)：[root@master2usr]#mysql-uroot-pGzdsj_1234gzdsj<gzdsj.sqlmysql:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure.檢查恢復(fù)結(jié)果：mysql:[Warning]Usingapasswordonthecommandlineinterfacecanbeinsecure.WelcometotheMySQLmonitor.Commandsendwith;or\g.YourMySQLconnectionidis971Copyright(c)2000,2019,Oracleand/oritsaffiliates.Allrightsreserved.OracleisaregisteredtrademarkofOracleCorporationand/oritsaffiliates.Othernamesmaybetrademarksoftheirrespectiveowners.Type'help;'or\h'forhelp.Type'\c'toclearthecurrentinputstatement.ReadingtableinformationforcompletionoftableandcolumnnamesYoucanturnoffthisfeaturetogetaquickerstartupwith-ADatabasechanged+ +|Tables_in_gzdsjl+ ++ +mysql>select*fromtfile_info;+++++++++3.4.5實(shí)驗(yàn)任務(wù)五：安裝和配置Redis[root@localhostopt]#tar-zxvfgcc.tar.gz-C/gcc/gcc/cpp-4.8.5-36.el7.x86_64.rpmgcc/gcc-c++-4.8.5-36.el7.x86_64.rpmgcc/glibc-headers-2.17-222.el7.x86_64.rpmgcc/kernel-devel-3.10.0-862.11.6.el7.x86_64.rpmgcc/kernel-headers-3.10.0-862.11.6.el7x86_64.rpmgcc/kernel-tools-3.10.0-862.11.6.el7.x86_64.rpmgcc/kernel-tools-libs-3.10.0-862.11.6.el7.x86_64.rpmgcc/kexec-tools-2.0.15-13.el7.x86_64.rpmgcc/libgcc-4.8.5-36.el7.x86_64.rpmgcc/libstdc++-4.8.5-36.el7.x86_64.rpmgcc/libstdc++-devel-4.8.5-36.el7.x86_64.rpmgcc/mpfr-3.1.1-4.el7.x86_64.rpm[root@localhostgcc]#rpm-Uvh*.rpm--nodeps--forcePreparing.#################################[100%]Updating/installing…1:libgcc-4.8.5-36.el7#################################[3%]2:glibc-common-2.17-222.el7#################################[7%]5:libmpc-1.0.1-3.el7#################################[17%]6:libstdc++-4.8.5-36.el7#################################[21%]7:libstdc++-devel-4.8.5-36.el7#################################[24%]9:kernel-tools-libs-3.10.0-862.11.6#################################[31%]10:libgomp-4.8.5-36.el7#################################[34%]11:kernel-headers-3.10.0-862.11.6.el#################################[38%]12:glibc-headers-2.17-222.el713:glibc-devel-2.17-222.el7#################################[45%]15:gcc-c++-4.8.5-36.el7#################################[52%]16:kernel-tools-3.10.0-862.11.6.el7#################################[55%]17:kexec-tools-2.0.15-13.el7#################################[59%]18:kernel-devel-3.10.0-862.11.6.el7#################################[62%]22:kernel-tools-3.10.0-957.12.2.el7#################################[76%]24:kernel-tools-libs-3.10.0-957.12.2#################################[83%]26:kexec-tools-2.0.15-21.el727:glibc-common-2.17-260.el7_6.5#################################[93%]#################################[100%][root@localhostgcc步驟四：上傳redis安裝包執(zhí)行以下命令解壓JDK安裝包：tar-zxvf/opt/redis-5.0.3.tar.gz-C/usr/localcd/usr/local/cd/usr/local/redis執(zhí)行以下命令編譯redis:執(zhí)行以下命令安裝redis:步驟五：修改redis配置將配置文件移動(dòng)到etc目錄mkdir/usr/local/redis/etcmv/usr/local/redis/redis.conf/usr/local/redis/etc/redis.conf修改配置文件，以守護(hù)縣城的形式運(yùn)行：vi/usr/local/redis/etc/redis.conf將bind注釋掉，表示允許通過所有接口地址訪問將文件中的daemonize改為yes,表示允許在后臺(tái)運(yùn)行將文件中requirepass配置開始，并將后面的字符串設(shè)置為需要設(shè)置的密碼，這里我們?cè)O(shè)置衛(wèi)gzdsj1234/usr/local/redis/bin/redis-server/usr/local/redis/etc/redis.conf/usr/local/redis/bin/redis-cli:6379>authgzdsj1234設(shè)置key的value::6379>getkey1:6379>delkey1(integer)1從List左邊添加一個(gè)或多個(gè)元素：(integer)2從List左邊出隊(duì)一個(gè)元素："value2"設(shè)置hash里面的一個(gè)字段值：(integer)1獲取hash里面字段值："valuel"添加一個(gè)或多個(gè)元素到集合：(integer)2獲取集合的所有元素：2)"value2*從集合刪除一個(gè)或多個(gè)元素：(integer)1步驟一：簡(jiǎn)單配置Redis/usr/local/redis/etc/redis.confdaemonizeyes#是否以daemorequirepassgzdsj1234#需要密碼訪問appendonlyno#是否開啟AOF持久化模式appendfilename"appendonly.aof"#AOF存儲(chǔ)文件名查看服務(wù)器信息：#Serverredis_version:5.0.3redis_git_sha1:00000000redis_git_dirty:0redis_build_id:73fb754199ecfef3redis_mode:standalonearch_bits:64multiplexing_api:epollatomicvar_api:atomic-builtinprocess_id:11946run_id:cc3100e8dd4a83306c272572c9ce4e53f6637be9tcp_port:6379uptime_in_seconds:608uptime_in_days:0configured_hz:10executable:/usr/local/redis/bin/redis-serverconfig_file:/usr/local/redis/etc/redis.conf#Clientsconnected_clients:1client_recent_max_input_buffer:2clientrecent_max_output_buffer:0blocked_clients:0#Memoryused_memory:912088used_memory_startup:861880allocator_allocated:878264allocator_active:21allocator_resident:2198528total_system_memory:16657154048maxmemory:0maxmemory_policy:noevictionallocator_frag_ratio:2.50allocator_frag_bytes:1320264mem_fragmentation_ratio:2.55mem_fragmentation_bytes:1358152mem_not_counted_for_evict:0mem_replication_backlog:0mem_allocator:libcactive_defrag_running:0#Persistenceloading:0aof_rewrite_scheduled:0aof_last_bgrewrite_stataof_lastcowsize:0#Statstotalconnectionsreceived:1total_commands_processinstantaneous_input_kbps:0.00instantaneous_output_kbps:0.00pubsub_patterns:0#Replicationrole:mastermaster_replid:b4034ad9b96bc748a4939d47b87aad782509114dmaster_replid2:0000000000000000000000000000000000000000second_repl_offset-1repl_backlog_active:0repl_backlog_histlen:0#CPU#Cluster#Keyspace異步保存數(shù)據(jù)到磁盤：Backgroundsavingstarted獲取上一次保存時(shí)間判斷保存操作是否成功：同步保存數(shù)據(jù)到磁盤：(integer)1清空當(dāng)前數(shù)據(jù)庫(kù)：3.4.8實(shí)驗(yàn)任務(wù)八：安裝和配置zookeeper(偽集群模式)步驟一：上傳zookeeper安裝包通過文件上傳工具將安裝文件zookeeper-3.4.14.tar.gz上傳到/opt目錄執(zhí)行以下命令創(chuàng)建安裝目錄：mkdir/usr/zookeeper執(zhí)行以下命令解壓JDK安裝包tar-zxf/opt/zookeeper-3.4.14.tar.gz-C/usr/zookeeper將/usr/zookeeper/zookeeper-3.4.14復(fù)制成三份：cd/usr/zookeepercp-R./zookeeper-3.4.14./zookeeper1cp-R./zookeeper-3.4.14./zookeeper2cp-R./zookeeper-3.4.14./zookeeper3cd/usr/zookeeper/zookeeper1/confvizoo.cfg修改dataDir配置：dataDir=/usr/zookeeper/zookeeper1/data添加集群配置：步驟四：配置zookeeper1節(jié)點(diǎn)的serverid創(chuàng)建data目錄：mkdir/usr/zookeeper/zookeeper1/data配置serverid:cd/usr/zookeeper/zookeeperl/dataecho'l'>myid步驟三：修改zookeeper2節(jié)點(diǎn)的配置文件編輯配置文件：cd/usr/zookeeper/zookeeper2/confvizoo.cfgclientPort=3181修改dataDir配置：dataDir=/usr/zookeeper/zookeeper2/data添加集群配置：步驟四：配置zookeeper2節(jié)點(diǎn)的serveridmkdir/usr/zookeeper/zookeeper2/datacd/usr/zookeeper/zookeeper2/dataecho'2'>myidcd/usr/zookeeper/zookeeper3/confcpzoo_sample.cfgzoo.cfgvizoo.cfg修改clientPort配置：clientPort=4181修改dataDir配置：dataDir=/usr/zookeeper/zookeeper3/data添加集群配置：步驟四：配置zookeeper3節(jié)點(diǎn)的serveridmkdir/usr/zookeeper/zookeeper3/datacd/usr/zookeeper/zookeeper3/dataecho'3'>myid步驟五：?jiǎn)?dòng)zookeeper執(zhí)行以下命令啟動(dòng)zookeeper集群：/usr/zookeeper/zookeeperl/bin/zkServer.shstart/usr/zookeeper/zookeeper2/bin/zkServer.shstart/usr/zookeeper/zookeeper3/bin/zkServer.shstart0步驟五：驗(yàn)證安裝結(jié)果cd/usr/zookeeper/zookeeper1/bin驗(yàn)證zookeeper基本操作：[zk:27:2181(CONNECTED)1]create/zktestmydataCreated/zktest/zktest[zk:27:2181(CONNECTED)3]get/zktestcZxid=0x1000000bdctime=SatMar0222:06:17EST2019mZxid=0x1000000bdpZxid=0x1000000bdcversion=0dataVersion=0aclVersion=0ephemeralOwner=0x0dataLength=6numChildren=0步驟一：上傳kafka安裝包通過文件上傳工具將安裝文件kafka_2.12-2.3.0.tgz上傳到/opt目錄步驟二：解壓安裝kafkatar-zxf/opt/kafka_2.12-2.3.0.tgz-C/usrcd/usrcd/usr/kafka/configviperties修改以下配置：isteners=PLAINTEXT://:9092新增節(jié)點(diǎn)2配置：cd/usr/kafka/configcppertiespertiesvipertiesbroker.id=1log.dirs=/tmp/kafka-logs-1新增節(jié)點(diǎn)3配置：cd/usr/kafka/config編輯配置文件：修改以下配置：broker.id=2log.dirs=/tmp/kafka-logs-2步驟六：?jiǎn)?dòng)kafka分別執(zhí)行以下命令啟動(dòng)各個(gè)節(jié)點(diǎn)：cd/usr/kafka/binsh./kafka-server-start.sh-daemon./config/perties&sh./kafka-server-start.sh-daemon./config/perties&sh./kafka-server-start.sh-daemon./config/perties&步驟七：驗(yàn)證安裝結(jié)果cd/usr/kafka/bin./kafka-topics.sh--create--bootstrap-servergzdjs-topic/kafka-topics.sh--describe--bootstrap-server執(zhí)行以下發(fā)送消息：/kafka-console-producer.sh執(zhí)行完進(jìn)入發(fā)送消息命令行，輸入需要發(fā)送的消息即可。執(zhí)行以下發(fā)送消息：執(zhí)行完可以看到控制臺(tái)接收到之前發(fā)送的消息。完成本實(shí)驗(yàn)，您應(yīng)該能夠：●掌握Hadoop的安裝部署●掌握Hive的安裝部署4.2實(shí)驗(yàn)要求●熟悉常用Linux操作系統(tǒng)命令本實(shí)驗(yàn)所需資源環(huán)境如表1-1所示。表1-1資源環(huán)境最低配置：雙核CPU、8GB內(nèi)存、100G硬盤CentOS-7-x86_64-Everything-1810大數(shù)據(jù)平臺(tái)hadoop-2.7.7、Hive-2.3.4jdk1.8.0_201、Hadoop-2.7.7、Hive-2.3.4、MySQL5.7.25步驟一：禁用防火墻停止防火墻：[root@localhost~]#systemctlstopfirewalld置禁用防火墻：禁用SELinux:[root@localhost~]#sed-i's/SELINUX=.*Loadedplugins:fastestmirrorLoadingmirrorspeedsfromcachedhostfile*base:mirror,*extras:*updates:extrasupdatesResolvingDependencies-->Runningtransactioncheck>Packagersyncx86_640:3.1.2-4.el7willbeinstalled-->FinishedDependencyResolution批注[wp2]:高版本的系統(tǒng)可能已經(jīng)存要重復(fù)安裝，檢測(cè)方法：rpm-qa|greprsyncDependenciesResolved=========================================================RepositorySize=========================ArchVersionArch===========================二：============TransactionSummary======================一=Totaldownloadsize:403kInstalledsize:815kDownloadingpackages:rsync-3.1.2-4.el7.x86_64.rpm|403RunningtransactioncheckRunningtransactiontestTransactiontestsucceededRunningtransactionVerifyingComplete!步驟四：安裝節(jié)點(diǎn)及目錄規(guī)劃節(jié)點(diǎn)類型IP地址主機(jī)名控制節(jié)點(diǎn)26masterl27master2數(shù)據(jù)節(jié)點(diǎn)2829客戶端節(jié)點(diǎn)30目錄規(guī)劃：●安裝包上傳目錄：/opt/hadoop配置并啟動(dòng)master1節(jié)點(diǎn)上的rsync服務(wù)：vi/etc/rsyncd.conf添加以下配置：[packages]path=/opt/hadoop/comment=hadoopinstallpackages[hadoop-config]path=/usr/hadoop/latest/etc/hadoop/comment=hadoopconfigfiles啟動(dòng)rsync服務(wù)：rsync--daemon在除master1以外的節(jié)點(diǎn)上執(zhí)行rsync同步命令，完成安裝包的拷貝mkdir/opt/hadooprsync-rmaster1:packages/opt/hadoop/在master1節(jié)點(diǎn)上生成相關(guān)密鑰及證書復(fù)雜公鑰及授權(quán)，以下命令在除master1以外的其它節(jié)點(diǎn)上執(zhí)行scproot@master1:~/.ssh/id_rsa.pub~cat~/id_rsa.pub>>~/.ssh/authorized_keyschmod600~/.ssh/authorized_keys執(zhí)行以下命令創(chuàng)建安裝目錄：mkdir/usr/hadoop執(zhí)行以下命令解壓Hadoop安裝包tar-zxf/opt/hadoop/hadoop-2.7.7.tar.gz-C/usr/hadoop執(zhí)行以下命令創(chuàng)建鏈接cd/usr/hadoop/In-s/usr/hadoop/hadoop-2.7.7/usr/hadoop/latest新增或編輯/etc/profile.d/hadoop.sh文件，添加以下內(nèi)容：執(zhí)行以下命令激活環(huán)境變量：source/etc/profile.d/hadoop.sh步驟四：準(zhǔn)備數(shù)據(jù)目錄執(zhí)行以下命令，準(zhǔn)備數(shù)據(jù)目錄：mkdir/var/hadoopmkdir/var/hadoop/namenodemkdir/var/hadoop/disk1mkdir/var/hadoop/disk2mkdir/var/hadoop/tmpmkdir/var/hadoop/logsvi/usr/hadoop/latest/etc/hadoop/core-site.xml文件<configuration></configuration>之間添加以下配置：<property><value>hdfs://master1:9000</val</property><property><name>hadoop.tmp.dir</name><value>/var/hadoop/tmp</value></property><property><name>xyuser.root.groups</name><value>*</value></property><property><name>xyuser.root.hosts</name><value>*</value></property>執(zhí)行以下命令新建/編輯hdfs-site.xml文件：vi/usr/hadoop/latest/etc/hadoop/hdfs-site.xml文件中添加以下配置：<property><value>/var/hadoop/namenode</value></property><propert