大數(shù)據(jù)集群配置過程-hive篇

大數(shù)據(jù)集群配置過程—hive篇JDDC_SEED_BIGDATA2015-01-15修改歷史修改日期擔(dān)當(dāng)者審閱人版本號新增2015-01-18孫云濤V1.01.概述本篇文檔主要講解hive的安裝過程。Hadoop是安裝hbase和hive的基礎(chǔ)，即安裝hbase和hive之前必須先安裝hadoop并且hdfs和mapreduce必須都功能正常。因為hbase和hive其底層所使用都是應(yīng)用hadoop的兩個核心部分—dfs和mapreduce。在安裝hadoop之前需要考濾操作系統(tǒng)的版本（32位還是64位）以及hbase和hive的版本，否則會引起jar包不支持而導(dǎo)致的jar包替換或jar包重新編譯等問題。Hadoop、hbase、hive的版本匹配信息如下：分布式平臺軟件選型（Hadoop）軟儺最新版（曷手）穩(wěn)定版三（推薦）（言方）穩(wěn)定版一（'呂方）Hadoop2.5.0'2.2.00.2HBase6.00.兆.50.90.3ZooKeeper.53.4.63.M攆Pig0.13.00.13.00.9.0Hive0.13.1、Chukwa0.5.00.5.0Q-4.0sqoop.2說明：紅色為必選。建議采用最新版由于我們所使用的操作系統(tǒng)centos6.5是32位，而且安裝的hadoop是hadoop2.2.0，所以本次安裝的hive版本是0.12.0切記，在安裝hbase之前一定先安裝hadoop，而且要確保hadoop中的HDFS和MAPREDUCE都是可以正常使用的。2.正文與hadoop和hbase不同，hive在安裝的過程中不需要把hive的包分別發(fā)布到個節(jié)點(diǎn)上，只需要在namenode節(jié)點(diǎn)上hive包進(jìn)行配置，但需要在配置的過程中指定各個datanode節(jié)點(diǎn)的主機(jī)名。2-1下載安裝mysql本次將Hive配置成Server模式，并且使用MySQL作為元數(shù)據(jù)數(shù)據(jù)庫。原則上MySQL不必要非得安裝在namenode節(jié)點(diǎn)上，可以裝在一個單獨(dú)的服務(wù)器上，并進(jìn)行遠(yuǎn)程聯(lián)接。本次技術(shù)檢證，由于資源有限，把mysql安裝在namenode節(jié)點(diǎn)上。下載MySQL-5.6.22-1.linux_glibc2.5.i386.rpm-bundle.tar，參考下載地址/downloads/。具體的安裝以及root初始密碼的修改請參考《linux下MySQL安裝及設(shè)置》和《MySQL修改root密碼的各種方法整理》這兩篇文檔。2.2創(chuàng)建Hive元數(shù)據(jù)庫創(chuàng)建數(shù)據(jù)庫hive：createdatabaseifnotexistshive;創(chuàng)建數(shù)據(jù)庫用戶hive：createuserhiveidentifiedby'hive2015';授權(quán)可以訪問數(shù)據(jù)庫hive的主機(jī)和用戶：grantallonhive.*to'hive'@'hadoop01'identifiedby'hive2015';grantallonhive.*to'hive'@'hadoop02'identifiedby'hive2015';grantallonhive.*to'hive'@'hadoop03'identifiedby'hive2015';grantallonhive.*to'hive'@'hadoop04'identifiedby'hive2015';2.3安裝以及配置hive下載hive-0.12.0-bin.tar.gz，參考下載地址/downloads.htmlo下載之后，把安裝包放在/home/hadoop/source/目錄下。[root@hadoop02~](cd/home/hadoop/source[root@hadoop02source]JIItotal938404-rw-r--r-rw-1—rdrwxr-xr-rw-1—rdrwxr-xr-rw-r--rdrwxr-xr-x-rw-r--r--Irwxrwxrwxdrwxr-xr-rw-r--r-rw-r--r-rw-1—rdrwxr-xr-rw-r--r-rw-r--r-rw-1—r-rw-1—r-vii-r—r-rw-r--r---rw-r--r---rw-r--r---rwxrwxrwx-rwxrwxrwx-rw-r--r---rwxrwxrwx-rw-r--r--111111018111011141111111111111rootroot67974root67974rootrootrootrootrootrootroot7155rootroot715571557155715571557155rootrootrootrootrootrootrootrootusersrootusersrootrootrootrootrootrootrootwheeIrootrootwheeIwheeIwheeIwheeIwheeIwheeIrootrootrootrootrootroot29662871409682738664409610922307340967936750415409665662469310026240228944524096381115341901381126954148805086523677675294229745247084463442463533441863DecDecDecDecJanJanJanJanJanJanJanJanNovOctJanNovNovNovNovNovNovDecJanJanJanJanJan282828247676242491424171424242424242425771471854configinfc.txtconfiginfc.txthadoop-2.0.0-aIphahadoop-2.0.0-aIpha.tar.gzhadoop-2.2.0hadoop-2.2.0.tar.gz06hbase-0.96.2-hadoop257hbase-0.96.2-hadoop2-bin.tar.gz25hive->hive-0.12.0-bin07hiE.1L0-bin13hi作-CL』-bin.tar~.空I10MySQL-5.6.22-1.Iinux_gIibc2.5.i386.rpm-bundle.tar24MySQL-client-5.6.22-1.Iinux_glibc2.5.i386.rPm05mysqI-conrector-java-S.1.34mysqI-connector-java-5.1.34.tar.gzMySQL-deveI-5.6.22-1.Iinux_gIibc2.5.i336.rpmMySQL-embedded-5.6.22-1.Iinux_glibc2.E.i38G.rpmMySQL-server-5.6.22-1.Iinux_gIibc2.5.i386.rpmMySQL-shared-5.6.22-1.1inux_gIibc2.5.i386.rpm26MySQL-shared-compat-5.6.22-1.Iinux_gIibc2.5.i386.rpm26MySQL-test-5.6.22-1.Iinuscglibc2.5.i386.rpm00profile.txtscp_hadoop.sh22scp_hbase.sh29scp_hive.shscp_profile.sh02zookeeper.out執(zhí)行以下命令：tarzxvfhive-0.12.0-bin.tar.gz執(zhí)行完成上述命令會么生一個hive-0.12.0-bin的目錄[rootWhadoLipU2~]1cd/home/hadoop/source[rootWhadrinpU2source]J11total938404-^|'|-r—-r--1rootroot2966Dec2815:54configinfc.txt-nrr—-r--1rootroot2871Dec2815:17configinfc.txtdr_,iiii:：＜r—:：＜r-:：■：:1167974users4096Dec2815:14hadoop-2.0.0-aIpha-^|'|-r—-r--1rootroot82738GG4Dec2414：18hadoop-2.0.0-aIpha.tar.gzdr_,iiii:：＜r—:：＜r-:：■：:1U67974users4096Jan714:42hadoop-2.2.0-nrr-T--1rootroot109223073Jan619:43hadoop-2.2.0.tar.gzdr_,iiii:：＜r—:：＜r-:：■：:0rootroot4096Jan715:06hbase-0.96.2-hadoop2-rwr--r--1rootroot79367504Jan619:57hbase-0.96.2-hadoop2-bin.tar.gz1rwxrwxrwx1rootroot15Jan2415:25hive-＞hive-。.12』-bindr'ivxr-xr-x10rootroot4叩5Jan2415；叮|hiv§-0.12』-bin1-r—-r--1rootroot65662469Jan915:13hive-0.12.0-bin.tar.gz-n/-r--r--1rootroot310026240Jan1414:10MySQL-5.6.22-1.1i1ibc2.5.i386.rpm-bund1e.tar-^|'|-r—-r--17155wheel22894452Nov2412:24MySQL-client-5.6.22-1.1inux_glibc2.5.i386.rPmdr—:：■：:r-:：■：:4routrout409HOct1714：05mysq1-cunnector-j5.1.34-r—-r--1rootroot3811153Jan1416:18mysq1-connector-iava-5.1.34.tar.gz-n/-r--r--17155wheel4190138Nov2412:24MySQL-deve1-5.6.22-1.1inux_g1ibc2.5.i336.rpm-^|'|-r—-r--17155wheel112695414Nov2412：25MySQL-embedded-5.6.22-1.1inux_g1ibc2.E.i386.rpm-^|'|-r—-r--17155wheel88050865Nov2412:25MySQL-server-5.6.22-1.1inux_g1ibc2.5.i386.rpm■削17155wheel2367767Nov2412:26MySQL-shared-5.g.22■1.1inu)cg1ibc2.5.i386.rpm-n/-r--r--17155wheel5294229Nov2412:26MySQL-shared-compat-5.6.22-1.1inux_g1ibc2.5.i386.rpm-rwr--r--17155wheel74524708Nov2412；26MySQL-test-5.6.22-1.1inuscglibc2.5.i386.rpm-^|'|-r—-r--1rootroot446Dec2510:00profile.txt-「"「'"「卅：：■::1rootroot344Jan713:41scp_hadoop.sh-■::1rootroot246Jan714:22scp_hbase.sh-^|'|-r—-r--1rootroot353Jan1416:29scp_hive.sh-1routrout344Jan713:42scp_prufile.sh-nrr—-r--1routrout1863Jan1816:02zuukeeper.outrii■創(chuàng)建hive包的聯(lián)接ln-s/home/hadoop/source/hive-0.12.0-bin/./hive在/etc/profile中新追加以下內(nèi)容exportHIVE_HOME=/home/hadoop/source/hiveexportPATH=$HIVE_HOME/bin:$PATH安裝MySQL-Connector，下載網(wǎng)址：/downloads/connector/。選擇“Connector/J”，接著選擇“PlatformIndependent"，本文下載的是“mysql-connector-java-5.1.30.tar.gz”。壓縮包“mysql-connector-java-5.1.30.tar.gz”中有個mysql-connector-java-5.1.30-bin.jar，解壓后將mysql-connector-java-5.1.30-bin.jar上傳到Hive的lib目錄下(/home/hadoop/source/hive-0.12.0-bin/lib/)，這個是MySQL的JDBC驅(qū)動程序。修改hive的配置文件(各配置文件位置：/home/hadoop/source/hive-0.12.0-bin/conf/)根據(jù)原來的幾個模板文件，生成hive需要使用的配置文件，執(zhí)行以下各個命令：perties修改hive-site.xml修復(fù)BUG該文件有個語法BUG，需要修改，進(jìn)入到hive-site.xml的第2000行，該行內(nèi)容為：<value>auth</auth>，明顯的語法錯誤，需要將“auth”改成“va山e”。修改javax.jdo.option.ConnectionURL將值設(shè)置為：jdbc:mysql://hadoop02:3306/hive?characterEncoding二UTF-8。修改javax.jdo.option.ConnectionDriverName將值設(shè)置為：com.mysql.jdbc.Driver。修改javax.jdo.option.ConnectionUserName將值設(shè)置為訪問hive數(shù)據(jù)庫的用戶名hive：<value>hive</value>。修改javax.jdo.option.ConnectionPassword將值設(shè)置為訪問hive數(shù)據(jù)庫的密碼：<value>hive2015</value>。修改hive.metastore.schema.verification該值試情況進(jìn)行修改。修改hive.zookeeper.quorum將值設(shè)置為：hadoop01,hadoop03,hadoop04ZooKeeper被安裝在這三臺機(jī)器上。修改hive.metastore.uris將值設(shè)置為：thrift://hadoop02:9083,9083為Hive元數(shù)據(jù)的RPC服務(wù)端口。修改hive.metastore.warehouse.dir將值設(shè)置為：/home/hadoop/source/hive/warehouse，注意啟動前，需要創(chuàng)建好該目錄(/home/hadoop/source/hive/warehouse)。修改hive.server2.thrift.bind.host設(shè)置為hadoop02。修改perties:修改日志文件存放目錄hive.log.dir=/home/hadoop/source/hive/logs如果沒有home/hadoop/source/hive/logs這個目錄，需要創(chuàng)建一下。修改perties:修改日志文件存放目錄hive.log.dir=/home/hadoop/source/hive/logs/exec如果沒有/home/hadoop/source/hive/logs/exec這個目錄，需要創(chuàng)建一下。2.4啟動運(yùn)行hive1）初始化metastore安裝配置好后，在啟動Hive服務(wù)端之前，需要在服務(wù)端執(zhí)行一次“schematool-dbTypemysql-initSchema”，以完成對metastore的初始化。2）啟動metastore執(zhí)行命令：hive--servicemetastore&3）啟動Hive服務(wù)執(zhí)行：hiveserver2&。4）進(jìn)入Hive命令行操作界面（類似于mysql）執(zhí)行：hive[rootOhidMpdlcorif]1hive15/01/2313:44:29[M=D16/01/2313沖;網(wǎng)[M=0[MTi15/Q1/2313沖：網(wǎng)[MT15/01/28胡沖灑[M=015/01/231七4ik網(wǎng)[hFD15/01/2313:44:28[M=DCtrifisjuniianadeprecbtian]CcEisuretlon,.Ion;Ccrifi(uratIan.depretatIan^Confiion.deprecblian：ConfIsureitlDn.dBPKBCEitlDn;Confi(urailan.depreEbtIanjCorifiwjurai.i[in-deprecbtian：rupr^di.imludii.recursivc-itdeF-recBteduinstebd.ruprodiKWi.irF^-riifiIe-ir^-utforiBtainputsdi-recursive和rx九峻扣iLwizeisdewEistHd,In就網(wǎng)thrupred.nin.spIit.￡ini￡depruated.Tnstcfid^rup「?[rootOhidMpdlcorif]1hive15/01/2313:44:29[M=D16/01/2313沖;網(wǎng)[M=0[MTi15/Q1/2313沖：網(wǎng)[MT15/01/28胡沖灑[M=015/01/231七4ik網(wǎng)[hFD15/01/2313:44:28[M=DCtrifisjuniianadeprecbtian]CcEisuretlon,.Ion;Ccrifi(uratIan.depretatIan^Confiion.deprecblian：ConfIsureitlDn.dBPKBCEitlDn;Confi(urailan.depreEbtIanjCorifiwjurai.i[in-deprecbtian：rupr^di.imludii.recursivc-itdeF-recBteduinstebd.ruprodiKWi.irF^-riifiIe-ir^-utforiBtainputsdi-recursive和rx九峻扣iLwizeisdewEistHd,In就網(wǎng)thrupred.nin.spIit.￡ini￡depruated.Tnstcfid^rup「?di.ninhs=>|jI,.ize-iF*riimdeprecblsduiwr&liiirir：3PHl,si2?.rer.rcrt：is：depr-ggeted.UBBwr豉也*irpjlJ|leirp.itIt,l6KSl禎userupr*l_E*_input.fileirputfc-nut.splILulnsiTEInslmd：,rupr^diKe-iir^-ulifile-irf-utfariblaS：Iidiin?iz?i.p?rbr-ickusirtt.ii..flleirp_tforget,$pIIt.ilngi曲「pemitarwpred.redjot-tisksis：deprecated.Tnstebd^usenfiprtdj?-jah.reductsrupr^durodLK^^l-isk?.iexecutianisdeprec^ilKluInsipid：.rAF-reduce.reduce^speculative查看有哪些表hishowtables>；UL_inviteepokesIImetaken：1.854seconds,Fetched：2rowts)hive>|注：invites和pokes這兩個表是在技術(shù)檢證時創(chuàng)建的，建表腳本如下：CREATETABLEpokes(fooINT,barSTRING)；CREATETABLEinvites(foINT,barSTRING)PARTITIONEDBY(dsSTRING);導(dǎo)入數(shù)據(jù)的語法如下：LOADDATALOCALINPATH'/home/hadoop/source/hive/examples/files/kv2.txt'OVERWRITEINTOTABLEinvitesPARTITION(ds=’2014');注：hive目前還不支持insertinto語法。查看表里的信息hive>SE-Ieci*frmirwitesvhe-refo=hive>SE-Ieci*frmirwitesvhe-refo='2^4-'；intelWepRaijceJobs-iLaunchIJobIaulof1bknberDfreduceta.skiissettoDsincethere'snoredjtearer&ioKCIsespsthaintalnsrultipleSLF^Jblrrilnss.SlfaJ:Fcurdbirdingin[Jar:f11e:/horie/r^iw/sftjrcei^dw-2.2.0/share/r(8idoDp/?riQriMIb/sIf4J-1j12-1□715.jarMors/sIf4j/1ifIZStaiIcLafeisrBincfer-cIesELF4J：Fojrdbindingin[Jh「葉ic!j^nrt/l-AdDflp|,l5oijKic?=/hiue-0.12.0-hinIib/aIf4-j-1a;4-j12-1.G.1.j-ar!/□rf/sIf4j/irpI/StiELoctTerBinir.clasa]洶http^//wv.3lf4J,orEr/co(fe3.htiril.llrijll：ipledblrrilnssfaranewpl^r^tIon,SLF垢IlrJbindintisof[ote.51f4J-iripI..Lds^jLoEE^rFaclory]IE/DI/2B13：^!4215/1?：柱:焰]efw「ins，IG/DI/2B13：iE：!4=!rins.15/DI/2B13:48:43IE/DI/2D13：1￡：43I5/DI/2B13：+E：4315/DI/2613:4B:4SIE/DI/2BI5/DI/2B13：+E：4315/0I/2B13:48:43EBttutionIo;di：yXlFtJutI.Hati,■.?eCJodd-abder!Unobl±holu.dnativc-hadoapihr&iyForj*aurpldiFam...usin;buiHin-jbvicvhereopplitable呻conf.GanflEureiIan：flIh：Aip7root/hi?_205-01-28LI3-41_139LJ821565161583B871812-1/-1ocft1-100D2/Jobconf.MillenattaiPttoawrrlVXlFfJtmf.Ekinficuratian!fiIE!/trp./?t/hi^2n15-0I-28_I3-4S-41_139_921565161593597IB12-1/-IotaI-IDOD2/jdKEof.Mil!anattenpttoautrri]NFOConFisLir^lionadBprKBition：INFOConfifijraticn.deprn&ca'Linn!]NFOCcnfiguneitIon.deprea&tinn：]NFOConFia-iadeprKBition!]NFOConficn.depKecbtinn!]NFOCcnfiguneitIon.dep