2023學(xué)年完整公開(kāi)課版Spark搭建部署

大數(shù)據(jù)平臺(tái)搭建和開(kāi)發(fā)學(xué)習(xí)計(jì)劃數(shù)據(jù)處理平臺(tái)搭建性能開(kāi)發(fā)性能檢索第一章大數(shù)據(jù)平臺(tái)搭建章節(jié)目錄1.簡(jiǎn)介

2.Hadoop平臺(tái)搭建3.MapReduce并行計(jì)算4.Spark系統(tǒng)該項(xiàng)需要在每個(gè)節(jié)點(diǎn)配置。通過(guò)scala可以進(jìn)行運(yùn)行Spark案例，首先對(duì)scala進(jìn)行安裝和配置。#yum-yinstallscala-2.11.7.rpm驗(yàn)證：#scala–version出現(xiàn)如下字樣Scalacoderunnerversion2.11.7--Copyright2002-2013,LAMP/EPFL該項(xiàng)需要在每個(gè)節(jié)點(diǎn)配置。通過(guò)scala可以進(jìn)行運(yùn)行Spark案例，首先對(duì)scala進(jìn)行安裝和配置。#yum-yinstallscala-2.11.7.rpm驗(yàn)證：#scala–version出現(xiàn)如下字樣Scalacoderunnerversion2.11.7--Copyright2002-2013,LAMP/EPFLMasterSpark系統(tǒng) Scala安裝與配置Slaver1&Slaver2該項(xiàng)需要在每個(gè)節(jié)點(diǎn)配置。進(jìn)入到軟件包所在的目錄/mnt/Rpms，使用yum命令安裝spark軟件包，進(jìn)入安裝目錄：#cd/mnt/Rpms/#yum-yinstallbigdata-xiandian-spark-1.0-1.x86_64.rpm#cd/opt/hadoop_resources/spark-1.5.2配置環(huán)境變量：#vi/etc/profileexportSPARK_HOME=/opt/hadoop_resources/spark-1.5.2exportPATH=$PATH:$SCALA_HOME/bin:$SPARK_HOME/bin使環(huán)境變量生效：#source/etc/profile該項(xiàng)需要在每個(gè)節(jié)點(diǎn)配置。進(jìn)入到軟件包所在的目錄/mnt/Rpms，使用yum命令安裝spark軟件包，進(jìn)入安裝目錄：#cd/mnt/Rpms/#yum-yinstallbigdata-xiandian-spark-1.0-1.x86_64.rpm#cd/opt/hadoop_resources/spark-1.5.2配置環(huán)境變量：#vi/etc/profileexportSPARK_HOME=/opt/hadoop_resources/spark-1.5.2exportPATH=$PATH:$SCALA_HOME/bin:$SPARK_HOME/bin使環(huán)境變量生效：#source/etc/profileMasterSpark系統(tǒng) Spark安裝Slaver1&Slaver2Master配置spark-env.sh#cdspark-1.5.2/conf/修改spark-env.sh文件#cpspark-env.sh.templatespark-env.sh編輯spark-env.sh添加以下內(nèi)容：exportJAVA_HOME=/usr/java/jdk1.7.0_67/exportSPARK_MASTER_IP=29exportSPARK_WORKER_MEMORY=1gexportHADOOP_CONF_DIR=/opt/hadoop_resources/hadoop-2.7.0/etc/hadoop修改slaves文件在master節(jié)點(diǎn)進(jìn)行配置#cpslaves.templateslaves編輯slaves，添加：masterslaver1slaver2Spark系統(tǒng) Spark配置Spark系統(tǒng) Spark配置

配置Hadoop環(huán)境變量在Yarn上運(yùn)行Spark需要配置HADOOP_CONF_DIR、YARN_CONF_DIR和HDFS_CONF_DIR環(huán)境變量，編輯/root/.bash_profile文件添加：exportHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexportHDFS_CONF_DIR=$HADOOP_HOME/etc/hadoopexportYARN_CONF_DIR=$HADOOP_HOME/etc/hadoop環(huán)境變量生效：#source/root/.bash_profileMasterSlaver1&Slaver2

配置Hadoop環(huán)境變量在Yarn上運(yùn)行Spark需要配置HADOOP_CONF_DIR、YARN_CONF_DIR和HDFS_CONF_DIR環(huán)境變量，編輯/root/.bash_profile文件添加：exportHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexportHDFS_CONF_DIR=$HADOOP_HOME/etc/hadoopexportYARN_CONF_DIR=$HADOOP_HOME/etc/hadoop環(huán)境變量生效：#source/root/.bash_profileSlaver1&Slaver2HadoopSlave節(jié)點(diǎn)上配置。接將HadoopMaster節(jié)點(diǎn)上的spark-1.5.2拷貝到HadoopSlave節(jié)點(diǎn)。#scp-rroot@master:/opt/hadoop_resources/spark-1.5.2/opt/hadoop_resources/Spark系統(tǒng) Spark配置在master節(jié)點(diǎn)進(jìn)行。#cd/opt/hadoop_