電商大數(shù)據(jù)分析項目部署運維手冊

電商大數(shù)據(jù)分析項目部署運維手冊XX科技股份有限公司編制

目錄一、文章編寫目的 5二、前置準(zhǔn)備 52.1配置網(wǎng)絡(luò)ip 52.2編寫相關(guān)操作腳本 81.批量執(zhí)行命令腳本 82.批量重命名腳本 93.批量拷貝腳本 104.cm_migrate.sh 125.format2.sh 126.mountDisk.sh 127.network.sh 138.node.list 149.node.txt 142.3hostname及hosts配置 141.配置每臺節(jié)點的hostname 142.配置節(jié)點ip、hostname映射 152.4禁用SELinux 152.5關(guān)閉防火墻 162.6設(shè)置swappiness 172.7關(guān)閉透明大頁面 182.8配置操作系統(tǒng)本地yum源 192.9安裝http服務(wù) 202.10集群時鐘同步 231.所有機器卸載chrony 232.所有機器安裝ntp 233.node1機器配置時鐘與自己同步 234.集群其它節(jié)點，配置找node1機器去同步 235.重啟所有機器的ntp服務(wù) 246.驗證始終同步，在所有節(jié)點執(zhí)行ntpq-p命令，如下使用腳本批量執(zhí)行 242.11安裝MariaDB 24三、ClouderaManager安裝 293.1配置本地CM源 293.2安裝ClouderaManagerServer 32四、CDH安裝 334.1CDH集群安裝向?qū)?334.2集群設(shè)置安裝向?qū)?40五、HBASE安裝 445.1準(zhǔn)備工作： 445.2安裝： 441.下載hbase1.3.2 442.解壓 443．配置環(huán)境變量~/.base_profile 444．hbase-env.sh配置java_home和ZK 455.hbase-site.xml配置參數(shù) 456．配置regionserver 467．scp-rhbase到其他節(jié)點 46六、Flume安裝 476.1安裝 47七、Kafka安裝: 477.1安裝 47

一、文章編寫目的本文檔主要描述Cloudera電商大數(shù)據(jù)分析平臺的安裝主要分為4個步驟：集群服務(wù)器配置，包括安裝操作系統(tǒng)、關(guān)閉防火墻、同步服務(wù)器時鐘等，又稱為集群安裝的前置準(zhǔn)備，關(guān)于更多前置條件準(zhǔn)備也可以參考之前的的文檔《CDH安裝前置準(zhǔn)備》；安裝ClouderaManager；通過ClouderaManager的WebUI向?qū)桨惭bCDH集群；集群完整性檢查，包括HDFS文件系統(tǒng)、MapReduce、Hive等是否可以正常運行。本文操作說明基于以下假設(shè)：操作系統(tǒng)版本：Redhat7.4CDH和CM版本均為5.16.1采用root對集群進行部署您已經(jīng)下載CDH和CM的安裝包二、前置準(zhǔn)備2.1配置網(wǎng)絡(luò)ip開啟虛擬機，修改配置文件輸入如下命令ll/etc/sysconfig/network-scripts/ifcfg-*會出現(xiàn)如下內(nèi)容（ifcfg-ens后面的名字可能不同）-rw.1rootroot6168月1319:25/etc/sysconfig/network-scripts/ifcfg-ens16777736

-rw-r--r--.1rootroot2541月32018/etc/sysconfig/network-scripts/ifcfg-lo輸入如下命令vi/etc/sysconfig/network-scripts/ifcfg-ens16777736出現(xiàn)下圖界面[root@cm1network-scripts]#viifcfg-eno16777736

#網(wǎng)絡(luò)類型（通常是Ethernet）

TYPE="Ethernet"

#IP的配置方法[none|static|bootp|dhcp]（引導(dǎo)時不使用協(xié)議|靜態(tài)分配IP|BOOTP協(xié)議|DHCP協(xié)議）

BOOTPROTO="dhcp"

DEFROUTE="yes"

IPV4_FAILURE_FATAL="no"

IPV6INIT="yes"

IPV6_AUTOCONF="yes"

IPV6_DEFROUTE="yes"

IPV6_FAILURE_FATAL="no"

NAME="eno16777736"

#隨機id

UUID="6d21f413-619d-4283-9ba2-e1a724c1c593"

DEVICE="eno16777736"

#系統(tǒng)啟動的時候網(wǎng)絡(luò)接口是否有效（yes/no）

ONBOOT="yes"

PEERDNS="yes"

PEERROUTES="yes"

IPV6_PEERDNS="yes"

IPV6_PEERROUTES="yes"

IPV6_PRIVACY="no"需要修改的地方有：BOOTPROTO=static#dhcp改為static（修改）

ONBOOT=yes#開機啟用本配置，一般在最后一行（修改）

IPADDR=01#靜態(tài)IP（增加）

GATEWAY=#默認(rèn)網(wǎng)關(guān)，虛擬機安裝的話，通常是2，也就是VMnet8的網(wǎng)關(guān)設(shè)置（增加）

NETMASK=#子網(wǎng)掩碼（增加）

DNS1=#DNS配置，虛擬機安裝的話，DNS就網(wǎng)關(guān)就行，多個DNS網(wǎng)址的話再增加（增加）這里注意，因為靜態(tài)ip地址設(shè)置為01，因此默認(rèn)網(wǎng)關(guān)和DNS地址前面部分,即192.168.139必須相同，不然會出現(xiàn)無法ping通的情況重啟網(wǎng)絡(luò)服務(wù)使ip生效servicenetworkrestart如果報錯,reboot重啟虛擬機三臺節(jié)點配置的ip分別為:31,32,33注意：每臺節(jié)點克隆后需要刪除每臺節(jié)點/etc/udev/rules.d/70-persistent-net.rules文件，清除mac地址。重啟每臺節(jié)點即可。節(jié)點克隆后還可以使用以下方式修改mac地址[root@hadoop101/]#vim/etc/udev/rules.d/70-persistent-net.rules進入如下頁面，刪除eth0該行；將eth1修改為eth0，同時復(fù)制物理ip地址，如圖所示修改ip地址[root@hadoop101/]#vim/etc/sysconfig/network-scripts/ifcfg-eth0

（1）把復(fù)制的物理ip地址更新

HWADDR=00:0C:2x:6x:0x:xx#MAC地址

（2）修改成你想要的ip

IPADDR=32#IP地址2.2編寫相關(guān)操作腳本腳本依賴expect程序，需要在執(zhí)行這些腳本的服務(wù)器上安裝expect,安裝方式：yuminstallexpect1.批量執(zhí)行命令腳本batch_cmd.sh#!/bin/sh

list_file=$1

cmd=$2

username=root

password=123456

cat$list_file|whilereadline

do

host_ip=`echo$line|awk'{print$1}'`

#username=`echo$line|awk'{print$2}'`

#password=`echo$line|awk'{print$3}'`

./expect_cmd$host_ip$username$password"$cmd"

doneexpect_cmd#!/usr/bin/expect

settimeout10

sethost[lindex$argv0]

setusername[lindex$argv1]

setpassword[lindex$argv2]

setcmd[lindex$argv3]

spawnssh-p22$username@$host"$cmd"

expect{

"(yes/no)?"

{

send"yes\n"

expect"*assword:"{send"$password\n"}

}

"*assword:"

{

send"$password\n"

}

}

expect"100%"

#expecteof

exit02.批量重命名腳本batchrenamecmd.sh#!/bin/sh

list_file=$1

username=root

password=123456

cat$list_file|whilereadline

do

host_ip=`echo$line|awk'{print$1}'`

host_name=`echo$line|awk'{print$2}'`

host_alias_name=`echo$line|awk'{print$3}'`

#username=`echo$line|awk'{print$2}'`

#password=`echo$line|awk'{print$3}'`

./expect_rename_cmd$host_ip$username$password$host_name

doneexpectrenamecmd#!/usr/bin/expect

settimeout10

sethost[lindex$argv0]

setusername[lindex$argv1]

setpassword[lindex$argv2]

sethostname[lindex$argv3]

spawnssh-p22$username@$host"sed-is/^HOST.*/HOSTNAME=$hostname//etc/sysconfig/network;hostname$hostname;hostnamectlset-hostname$hostname"

expect{

"(yes/no)?"

{

send"yes\n"

expect"*assword:"{send"$password\n"}

}

"*assword:"

{

send"$password\n"

}

}

expect"100%"

#expecteof

exit03.批量拷貝腳本batch_scp.sh#!/bin/sh

list_file=$1

src_file=$2

dest_file=$3

username=root

password=123456

cat$list_file|whilereadline

do

host_ip=`echo$line|awk'{print$1}'`

#username=`echo$line|awk'{print$2}'`

#password=`echo$line|awk'{print$3}'`

./expect_scp$host_ip$username$password$src_file$dest_file

doneexpect_scp#!/usr/bin/expect

settimeout10

sethost[lindex$argv0]

setusername[lindex$argv1]

setpassword[lindex$argv2]

setsrc_file[lindex$argv3]

setdest_file[lindex$argv4]

spawnscp-r$src_file$username@$host:$dest_file

expect{

"(yes/no)?"

{

send"yes\n"

expect"*assword:"{send"$password\n"}

}

"*assword:"

{

send"$password\n"

}

}

expect"100%"

expecteof4.cm_migrate.sh#!/bin/bash

#updateCMIP

if[$#!=1];then

echo"USAGE:$0<CM_IP>"

echo"e.g.:$0"

exit1;

fi

shbatch_cmd.shnode.list"sed-is/^server_host=.*/server_host=$1//etc/cloudera-scm-agent/config.ini"

#updatemysql

echo"updateCMHOSTID"

host_id=`mysql-uroot-p123456--execute="usecm;selectHOST_IDfromHOSTSwhereIP_ADDRESS='$1';"`

host_id2=$(echo${host_id}|awk-F'''{print$2}')

#echo${host_id}

#echo${host_id2}

mysql-uroot-p123456--execute="usecm;updateROLESsetHOST_ID=${host_id2}whereNAMElike'mgmt-%';"5.format2.sh#!/bin/sh

foriin{b..m}

do

parted-s/dev/sd$imklabelgpt

parted-s/dev/sd$imkpartprimary2048s100%

/usr/sbin/mkfs.xfs-f/dev/sd${i}1

done6.mountDisk.sh#!/bin/sh

#backup/etc/fstab

cp/etc/fstab/etc/fstab.bak

PARTITION_LIST="sdb1sdc1sdd1sde1sdf1sdg1sdh1sdi1sdj1sdk1sdl1sdm1"

i=1

forPARTITIONin$PARTITION_LIST

do

UUID=`blkid"/dev/""$PARTITION"|awk'{print$2}'|sed's/\"http://g'`

echo$UUID

echo"add$PARTITIONto/etc/fstab"

MOUNTDIR="/data""$i"

i=$((i+1))

echo"mkdir-p$MOUNTDIR"

mkdir-p$MOUNTDIR

echo"appending\"$UUID$MOUNTDIRxfsdefaults00\"to/etc/fstab"

echo"$UUID$MOUNTDIRxfsdefaults00">>/etc/fstab

echo""

done

#mountallpartitions

mount-a

#showmountedpartitions

df-work.shvirshnet-destroydefault

virshnet-undefinedefault

systemctlrestartlibvirtd.service

yumremovelibvirt8.node.list31

32

339.node.txt31 node1

32 node2

33 node32.3hostname及hosts配置集群中各個節(jié)點之間能互相通信使用靜態(tài)IP地址。IP地址和主機名通過/etc/hosts配置，主機名通過/etc/hostname進行配置。1.配置每臺節(jié)點的hostname/etc/hostname文件如下：node1或者可以通過命令修改立即生效[root@ip-172-31-13-38~]$hostnamectlset-hostnamenode1可以使用腳本執(zhí)行[root@localhostbin]#shbatch_rename_cmd.shnode.txt

spawnssh-p22root@31sed-is/^HOST.*/HOSTNAME=//etc/sysconfig/network;hostname;hostnamectlset-hostname

root@31'spassword:

spawnssh-p22root@32sed-is/^HOST.*/HOSTNAME=//etc/sysconfig/network;hostname;hostnamectlset-hostname

root@32'spassword:

spawnssh-p22root@33sed-is/^HOST.*/HOSTNAME=//etc/sysconfig/network;hostname;hostnamectlset-hostname

root@33'spassword:2.配置節(jié)點ip、hostname映射在每臺節(jié)點/etc/hosts文件后追加如下內(nèi)容，配置ip與hostname之間的映射31 node1

32 node2

33 node3可以將node.txt文件內(nèi)容追加到/etc/hostscatnode.txt>>/etc/hosts腳本執(zhí)行shbatch_scp.shnode.list/etc/hosts/etc/2.4禁用SELinuxSELinux就是Security-EnhancedLinux的簡稱，安全加強的linux。傳統(tǒng)的linux權(quán)限是對文件和目錄的owner,group和other的rwx進行控制，而SELinux采用的是委任式訪問控制，也就是控制一個進程對具體文件系統(tǒng)上面的文件和目錄的訪問，SELinux規(guī)定了很多的規(guī)則，來決定哪個進程可以訪問哪些文件和目錄。雖然SELinux很好用，但是在多數(shù)情況我們還是將其關(guān)閉，因為在不了解其機制的情況下使用SELinux會導(dǎo)致軟件安裝或者應(yīng)用部署失敗。在所有節(jié)點執(zhí)行setenforce0命令shbatch_cmd.shnode.list"setenforce0"集群所有節(jié)點修改/etc/selinux/config文件如下(重啟才能生效,所有需要配置上面的臨時關(guān)閉)：SELINUX=disabled

SELINUXTYPE=targeted

shbatch_cmd.shnode.list"sed-is/^SELINUX=.*/SELINUX=disabled//etc/selinux/config"批量拷貝到其他節(jié)點shbatch_scp.shnode.list/etc/selinux/config/etc/selinux/檢查所有機器是否已修改成功：shbatch_cmd.shnode.list"cat/etc/selinux/config"|grepSELINUX2.5關(guān)閉防火墻集群所有節(jié)點執(zhí)行systemctlstopfirewalld命令shbatch_cmd.shnode.list"systemctlstopfirewalld"

shbatch_cmd.shnode.list"systemctldisablefirewalld"

shbatch_cmd.shnode.list"systemctlstatusfirewalld"|grepActive2.6設(shè)置swappiness所有節(jié)點執(zhí)行shbatch_cmd.shnode.list"sysctlvm.swappiness=1"

shbatch_cmd.shnode.list"echo1>/proc/sys/vm/swappiness"

shbatch_cmd.shnode.list"cat/proc/sys/vm/swappiness"2.7關(guān)閉透明大頁面在管理節(jié)點使用腳本執(zhí)行：shbatch_cmd.shnode.list"echonever>/sys/kernel/mm/transparent_hugepage/defrag"

shbatch_cmd.shnode.list"cat/sys/kernel/mm/transparent_hugepage/defrag"

shbatch_cmd.shnode.list"echonever>/sys/kernel/mm/transparent_hugepage/enabled"

shbatch_cmd.shnode.list"cat/sys/kernel/mm/transparent_hugepage/enabled"設(shè)置開機自關(guān)閉將如下腳本添加到/etc/rc.d/rc.local文件中iftest-f/sys/kernel/mm/transparent_hugepage/enabled;then

echonever>/sys/kernel/mm/transparent_hugepage/enabled

fi

iftest-f/sys/kernel/mm/transparent_hugepage/defrag;then

echonever>/sys/kernel/mm/transparent_hugepage/defrag

fi也可直接使用腳本shbatch_cmd.shnode.txt'echo"iftest-f/sys/kernel/mm/transparent_hugepage/enabled;then">>/etc/rc.d/rc.local;echo"echonever>/sys/kernel/mm/transparent_hugepage/enabled">>/etc/rc.d/rc.local;echo"fi">>/etc/rc.d/rc.local'

shbatch_cmd.shnode.txt'echo"iftest-f/sys/kernel/mm/transparent_hugepage/defrag;then">>/etc/rc.d/rc.local;echo"echonever>/sys/kernel/mm/transparent_hugepage/defrag">>/etc/rc.d/rc.local;echo"fi">>/etc/rc.d/rc.local'同步到所有節(jié)點#賦予rc.local腳本可執(zhí)行權(quán)限

cat/etc/rc.d/rc.local

shbatch_scp.shnode.list/etc/rc.d/rc.local/etc/rc.d/

shbatch_cmd.shnode.list"chmod+x/etc/rc.d/rc.local"

shbatch_cmd.shnode.list"ls-l/etc/rc.d|greprc.local"2.8配置操作系統(tǒng)本地yum源掛載操作系統(tǒng)iso文件mkdir-p/media/DVD1

mount-oloopCentOS-7-x86_64-DVD-1511.iso/media/DVD1/

df-h配置操作系統(tǒng)repo[root@node1~]$sudovi/etc/yum.repos.d/local_os.repo

[local_iso]

name=CentOS-$releasever-Media

baseurl=file:///media/DVD1

gpgcheck=0

enabled=1

[root@node1~]$sudoyumrepolist以上內(nèi)容中，baseurl是指Yum源的地址，該路徑下有個repodata目錄，就是yum安裝源目錄。file://表示yum源為文件。如果只想讓local.repo生效，可以在每臺節(jié)點的/etc/yum.repos.d目錄下創(chuàng)建一個repobak目錄，將其他的以“.repo”結(jié)尾的文件都移動到repobak目錄中。shbatch_cmd.shnode.list"mkdir-p/etc/yum.repos.d/repo_bak"

shbatch_cmd.shnode.list"mv/etc/yum.repos.d/CentOS-*/etc/yum.repos.d/repo_bak/"在每臺節(jié)點上執(zhí)行以下命令，更新yum源shbatch_cmd.shnode.list"yumcleanall"

shbatch_cmd.shnode.list"yumrepolist"2.9安裝http服務(wù)安裝apache的httpd服務(wù)主要用于搭建OS、ClouderaManager、CDH的yum源。在集群服務(wù)器中選擇一臺服務(wù)器來安裝httpd服務(wù)，命令如下：安裝httpd服務(wù)yum-yinstallhttpd啟動httpd服務(wù)systemctlstarthttpd

systemctlstatushttpd

systemctlenablehttpd安裝完httpd后，重新制作操作系統(tǒng)repo，換成http的方式方便其它服務(wù)器也可以訪問mkdir/var/www/html/iso

scp-r/media/DVD1/*/var/www/html/iso/

[root@node1~]$sudovi/etc/yum.repos.d/os.repo

[osrepo]

name=os_repo

baseurl=01/iso/

enabled=true

gpgcheck=false

[root@node1~]$sudoyumrepolist在瀏覽器輸入01/iso看到如下截圖則說明啟動成功。注：需要將os.repo文件同步至集群所有服務(wù)器的/etc/yum.repo.d目錄。shbatch_scp.shnode.list/etc/yum.repos.d/os.repo/etc/yum.repos.d/2.10集群時鐘同步在Redhat7.x的操作系統(tǒng)上，已經(jīng)默認(rèn)的安裝了chrony，我們這里先卸載chrony，然后安裝ntp。使用ntp來配置各臺機器的時鐘同步，將node1（31）服務(wù)作為本地ntp服務(wù)器，其它2臺服務(wù)器與其保持同步。1.所有機器卸載chronyshbatch_cmd.shnode.list"yum-yremovechrony"2.所有機器安裝ntpshbatch_cmd.shnode.list"yum-yinstallntp"3.node1機器配置時鐘與自己同步[root@node1~]#vi/etc/ntp.conf

#server0.iburst

#server1.iburst

#server2.iburst

#server3.iburst

server#localclock

fudgestratum104.集群其它節(jié)點，配置找node1機器去同步#server0.iburst

#server1.iburst

#server2.iburst

#server3.iburst

server015.重啟所有機器的ntp服務(wù)shbatch_cmd.shnode.list"systemctlstartntpd"

shbatch_cmd.shnode.list"systemctlstatusntpd"|grepActive

shbatch_cmd.shnode.list"systemctlenablentpd"6.驗證始終同步，在所有節(jié)點執(zhí)行ntpq-p命令，如下使用腳本批量執(zhí)行shbatch_cmd.shnode.list"ntpq-p"左邊出現(xiàn)*號表示同步成功。2.11安裝MariaDB安裝MariaDByum-yinstallmariadb

yum-yinstallmariadb-server啟動并配置MariaDBsystemctlstartmariadb

systemctlstatusmariadb

systemctlenablemariadb

[root@node1~]#/usr/bin/mysql_secure_installation

NOTE:RUNNINGALLPARTSOFTHISSCRIPTISRECOMMENDEDFORALLMariaDB

SERVERSINPRODUCTIONUSE!PLEASEREADEACHSTEPCAREFULLY!

InordertologintoMariaDBtosecureit,we'llneedthecurrent

passwordfortherootuser.Ifyou'vejustinstalledMariaDB,and

youhaven'tsettherootpasswordyet,thepasswordwillbeblank,

soyoushouldjustpressenterhere.

Entercurrentpasswordforroot(enterfornone):#回車

OK,successfullyusedpassword,movingon...

SettingtherootpasswordensuresthatnobodycanlogintotheMariaDB

rootuserwithouttheproperauthorisation.

Setrootpassword?[Y/n]Y

Newpassword:

Re-enternewpassword:

Passwordupdatedsuccessfully!

Reloadingprivilegetables..

...Success!

Bydefault,aMariaDBinstallationhasananonymoususer,allowinganyone

tologintoMariaDBwithouthavingtohaveauseraccountcreatedfor

them.Thisisintendedonlyfortesting,andtomaketheinstallation

goabitsmoother.Youshouldremovethembeforemovingintoa

productionenvironment.

Removeanonymoususers?[Y/n]Y

...Success!

Normally,rootshouldonlybeallowedtoconnectfrom'localhost'.This

ensuresthatsomeonecannotguessattherootpasswordfromthenetwork.

Disallowrootloginremotely?[Y/n]n

...skipping.

Bydefault,MariaDBcomeswithadatabasenamed'test'thatanyonecan

access.Thisisalsointendedonlyfortesting,andshouldberemoved

beforemovingintoaproductionenvironment.

Removetestdatabaseandaccesstoit?[Y/n]Y

-Droppingtestdatabase...

...Success!

-Removingprivilegesontestdatabase...

...Success!

Reloadingtheprivilegetableswillensurethatallchangesmadesofar

willtakeeffectimmediately.

Reloadprivilegetablesnow?[Y/n]Y

...Success!

Cleaningup...

Alldone!Ifyou'vecompletedalloftheabovesteps,yourMariaDB

installationshouldnowbesecure.

ThanksforusingMariaDB!建立CM，Hive等需要的表[root@node1~]#mysql-uroot-p

Enterpassword:

WelcometotheMariaDBmonitor.Commandsendwith;or\g.

YourMariaDBconnectionidis9

Serverversion:5.5.56-MariaDBMariaDBServer

Copyright(c)2000,2017,Oracle,MariaDBCorporationAbandothers.

Type'help;'or'\h'forhelp.Type'\c'toclearthecurrentinputstatement.

MariaDB[(none)]>

createdatabasemetastoredefaultcharactersetutf8;

CREATEUSER'hive'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONmetastore.*TO'hive'@'%';

FLUSHPRIVILEGES;

createdatabasecmdefaultcharactersetutf8;

CREATEUSER'cm'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONcm.*TO'cm'@'%';

FLUSHPRIVILEGES;

createdatabaseamdefaultcharactersetutf8;

CREATEUSER'am'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONam.*TO'am'@'%';

FLUSHPRIVILEGES;

createdatabasermdefaultcharactersetutf8;

CREATEUSER'rm'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONrm.*TO'rm'@'%';

FLUSHPRIVILEGES;

createdatabasehuedefaultcharactersetutf8;

CREATEUSER'hue'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONhue.*TO'hue'@'%';

FLUSHPRIVILEGES;

createdatabaseooziedefaultcharactersetutf8;

CREATEUSER'oozie'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONoozie.*TO'oozie'@'%';

FLUSHPRIVILEGES;

createdatabasesentrydefaultcharactersetutf8;

CREATEUSER'sentry'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONsentry.*TO'sentry'@'%';

FLUSHPRIVILEGES;

createdatabasenav_msdefaultcharactersetutf8;

CREATEUSER'nav_ms'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONnav_ms.*TO'nav_ms'@'%';

FLUSHPRIVILEGES;

createdatabasenav_asdefaultcharactersetutf8;

CREATEUSER'nav_as'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONnav_as.*TO'nav_as'@'%';

FLUSHPRIVILEGES;安裝jdbc驅(qū)動mkdir-p/usr/share/java/

mvmysql-connector-java-5.1.34.jar/usr/share/java/

cd/usr/share/java

ln-smysql-connector-java-5.1.34.jarmysql-connector-java.jar

[root@node1java]#ll

total940

-rwxrwxr-x.1rootroot960372Feb108:31mysql-connector-java-5.1.34.jar

lrwxrwxrwx1rootroot31Feb200:52mysql-connector-java.jar->mysql-connector-java-5.1.34.jar三、ClouderaManager安裝3.1配置本地CM源下載CM5.16.1的安裝包，地址為：/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/enterprise-debuginfo-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/jdk-6u31-linux-amd64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm下載CDH5.16.1的安裝包，地址為：/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel

/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1

/cdh5/parcels/5.16.1/manifest.json將ClouderaManager安裝需要的7個rpm包下載到本地，放在同一目錄，執(zhí)行createrepo命令生成rpm元數(shù)據(jù)。[root@node1cm5.16.1]#ll

total1019160

-rw-r--r--1rootroot9864584Nov2714:40cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot789872988Nov2714:40cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot8704Nov2714:40cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot10612Nov2714:40cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot30604172Nov2714:40enterprise-debuginfo-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot71204325Nov2714:40jdk-6u31-linux-amd64.rpm

-rw-r--r--1rootroot142039186Nov2714:40oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

[root@node1cm5.16.1]#createrepo.

Spawningworker0with2pkgs

Spawningworker1with2pkgs

Spawningworker2with2pkgs

Spawningworker3with1pkgs

WorkersFinished

SavingPrimarymetadata

Savingfilelistsmetadata

Savingothermetadata

GeneratingsqliteDBs

SqliteDBscomplete如果createrepo命令沒有,使用yum下載配置Web服務(wù)器將上述cdh5.16/cm5.16目錄移動到/var/www/html目錄下,使得用戶可以通過HTTP訪問這些rpm包。[root@node1~]#mvcm5.16/cdh5.16//var/www/html/驗證瀏覽器能否正常訪問制作ClouderaManager的repo源[root@node1~]#vi/etc/yum.repos.d/cm.repo

[cmrepo]

name=cm_repo

baseurl=01/cm5.16.2

enable=true

gpgcheck=false

[root@node1yum.repos.d]#yumrepolist

Loadedplugins:amazon-id,rhui-lb,search-disabled-repos

repoidreponamestatus

cmrepocm_repo7

rhui-REGION-client-config-server-7/x86_64RedHatUpdateInfrastructure2.0ClientC1

rhui-REGION-rhel-server-releases/7Server/x86_64RedHatEnterpriseLinuxServer7(RPMs)20,668

rhui-REGION-rhel-server-rh-common/7Server/x86_64RedHatEnterpriseLinuxServer7RHCommo233

repolist:20,909驗證安裝JDKyum-yinstalloracle-j2sdk1.7-1.7.0+update67-13.2安裝ClouderaManagerServer通過yum安裝ClouderaManagerServeryum-yinstallcloudera-manager-server初始化數(shù)據(jù)庫[root@node1yum.repos.d]#/usr/share/cmf/schema/scm_prepare_database.shmysqlcmcmpassword

JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

Verifyingthatwecanwriteto/etc/cloudera-scm-server

CreatingSCMconfigurationfilein/etc/cloudera-scm-server

Executing:/usr/java/jdk1.7.0_67-cloudera/bin/java-cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/*com.cloudera.enterprise.dbutil.DbCommandExecutor/etc/cloudera-scm-server/pertiescom.cloudera.cmf.db.

[main]DbCommandExecutorINFOSuccessfullyconnectedtodatabase.

Alldone,yourSCMdatabaseisconfiguredcorrectly!

6.2版本命令

/opt/cloudera/cm/schema/scm_prepare_database.shmysqlcmcmpassword啟動ClouderaManagerServersystemctlstartcloudera-scm-server檢查端口是否監(jiān)聽netstat-lnpt|grep7180出現(xiàn)以上情況,使用以下命令yum-yinstallnet-tools通過01:7180/cmf/login訪問CM四、CDH安裝4.1CDH集群安裝向?qū)dmin/admin登錄到CM同意license協(xié)議，點擊繼續(xù)選擇60試用，點擊繼續(xù)點擊“繼續(xù)”輸入主機ip或者名稱，點擊搜索找到主機后點擊繼續(xù)點擊“繼續(xù)”注意這里列出來的CDH版本都是系統(tǒng)最開始默認(rèn)的，來自于Cloudera公網(wǎng)的下載源倉庫，這里我們需要先將CDH的安裝源修改一下。使用parcel選擇，點擊“更多選項”,點擊“-”刪除其它所有地址，輸入01/cdh5.16.2，點擊“保存更改”保存更改后，這時回到上個頁面會看到我們之前準(zhǔn)備好的http的CDH5.16.1的源，如果顯示不出來，可能http源配置有問題，請參考前面步驟仔細(xì)進行檢查。選擇自定義存儲庫，輸入cm的http地址點擊“繼續(xù)”，進入下一步安裝jdk點擊“繼續(xù)”，進入下一步，默認(rèn)多用戶模式，不需要進行任何勾選點擊“繼續(xù)”，進入下一步配置ssh賬號密碼點擊“繼續(xù)”，進入下一步，安裝ClouderaManager相關(guān)到各個節(jié)點如果安裝速度很慢,注意檢查本地yum源是否已經(jīng)配置好點擊“繼續(xù)”，進入下一步安裝cdh到各個節(jié)點點擊“繼續(xù)”，進入下一步主機檢查，確保所有檢查項均通過點擊完成進入服務(wù)安裝向?qū)А?.2集群設(shè)置安裝向?qū)нx擇需要安裝的服務(wù)點擊“繼續(xù)”，進入集群角色分配點擊“繼續(xù)”，進入下一步，測試數(shù)據(jù)庫連接測試成功，點擊“繼續(xù)”，進入目錄設(shè)置，此處使用默認(rèn)默認(rèn)目錄，根據(jù)實際情況進行目錄修改點擊“繼續(xù)”，進入各個服務(wù)啟動安裝成功后進入CM管理界面主頁五、HBASE安裝5.1準(zhǔn)備工作：不同機器之間的時間同步要求每個節(jié)點子在30秒[root@hadoopNode5~]#yum-yinstallntp

#安裝ntp軟件

[root@hadoopNode5~]#ntpdate//指定與啊里云時間同步服務(wù)5.2安裝：1.下載hbase1.3.22.解壓[ambow@hadoopNode1~]$tar-xvzf~/soft/hbase-1.3.2-bin.tar.gz-C~/app/3．配置環(huán)境變量~/.base_profileHBASE_HOMEPATH4．hbase-env.sh配置java_home和ZK[ambow@hadoopNode1conf]$vi

$HBASE_HOME/conf/hbase-env.sh?

exportJAVA_HOME=/home/wl/app/jdk1.8.0_121

exportHADOOP_HOME=/home/wl/app/hadoop-2.7.3

?

exportHBASE_MANAGES_ZK=false

#禁用Hbase使用內(nèi)置zookeper

?

exportHBASE_BACKUP_MASTERS=${HBASE_HOME}/conf/backup-masters#配置HA的第二個節(jié)HMaster節(jié)點

新建一個$HBASE_HOME/conf/backup-masters文件vi$HBASE_HOME/conf/backup-masters把備用的HMaster節(jié)點添加:hadoopNode25.hbase-site.xml配置參數(shù)?

<configuration>

<!--#指定hbase在HDFS中目錄自動創(chuàng)建-->

<property>

<name>hbase.rootdir</name>

<value>hdfs://clusterldg/hbase</value>

</property>

<!--#true時，為集群模式u-->

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

<!--#設(shè)置自己的zookeeper用的那個幾個節(jié)點-->

<property>

<name>hbase.zookeeper.quorum</name>

<value>hadoopNode1,hadoopNode2,hadoopNode3,hadoopNode4,hadoopNode5</value>

</property>

?

?

<!--#使用內(nèi)置Zookeeper時要指定-->

<property>

<name>perty.dataDir</name>

<value>/home/ambow/zkdata/hdata</value>

</property>

</configuration>

?6．配置regionserver（配置每一個機器名子節(jié)點名不要配主節(jié)名）在hbase/conf/下新建regionserver文件,添加如入內(nèi)容hadoopNode3

hadoopNode4

hadoopNode5

?7．scp-rhbase到其他節(jié)點[ambow@hadoopNode1conf]$scp-r~/app/hbase-1.3.2

ambow@hadoopNode5:~/app/

[ambow@hadoopNode1conf]$scp-r~/app/hbase-1.3.2

ambow@hadoopNode4:~/app/

[ambow@hadoopNode1conf]$scp-r~/app/hbase-1.3.2

ambow@had