陕西网站建设公司找哪家,郑州网课老师,婚纱网站策划书模板,校园网站建设多少钱文章目录一、设计集群图二、准备五台虚拟机2.1、下载安装文件2.2、创建虚拟机2.3、配置网络2.4、修改主机名称2.5、关闭防火墙2.6、同步时间2.7、设置/etc/hosts文件2.8、设置免密登录2.9、为后面可以主备替换安装psmisc三、安装JDK3.1、安装jdk3.2、测试jdk是否安装成功3.3、将…
文章目录一、设计集群图二、准备五台虚拟机2.1、下载安装文件2.2、创建虚拟机2.3、配置网络2.4、修改主机名称2.5、关闭防火墙2.6、同步时间2.7、设置/etc/hosts文件2.8、设置免密登录2.9、为后面可以主备替换安装psmisc三、安装JDK3.1、安装jdk3.2、测试jdk是否安装成功3.3、将配置文件和安装目录传输给其他主机四、搭建zookeeper集群3.1、安装zookeeper3.2、将配置文件和安装目录传输给其他主机3.3、修改其他主机的id3.4、集群使用脚本3.5、测试zookeeper是否安装成功四、搭建hadoop集群4.1、安装hadoop4.2、core-site.xml4.3、hadoop-env.sh4.4、hdfs-site.xml4.5、mapred-site.xml4.6、yarn-site.xml4.7、workers4.8、/etc/profile4.9、将配置文件和安装目录传输给其他主机五、集群首次启动六、测试集群是否成功6.1、上传文件测试6.2使用jar包运行测试一、设计集群图 关于JournalNode 一探究竟Namenode、SecondaryNamenode、NamenodeHA关系 DFSZKFailoverController 高可用时它负责监控NN的状态并及时的把状态信息写入ZK。它通过一个独立线程周期性的调用NN上的一个特定接口来获取NN的健康状态。FC也有选择谁作为Active NN的权利因为最多只有两个节点目前选择策略还比较简单先到先得轮换。 二、准备五台虚拟机
2.1、下载安装文件 链接https://pan.baidu.com/s/19s19t-yDcxRuj0AnnDOSkQ 提取码n7km 需要下方红框的三个文件 2.2、创建虚拟机 创建centos虚拟机 2.3、配置网络 配置网络 2.4、修改主机名称 例如我主机设置ip为192.168.136.31-35将虚拟机名称修改为xsqone31-35 192.168.136.41hostnamectl set-hostname xsqone31
192.168.136.42hostnamectl set-hostname xsqone32
192.168.136.43hostnamectl set-hostname xsqone33
192.168.136.44hostnamectl set-hostname xsqone34
192.168.136.45hostnamectl set-hostname xsqone35
使用bash命令可以即可生效。 2.5、关闭防火墙 5台机器都需要执行命令 [rootxsqone31 ~]# systemctl stop firewalld
[rootxsqone31 ~]# systemctl disable firewalld2.6、同步时间 5台机器都需要执行命令 # 同步时间[rootxsqone31 ~]# yum install -y ntpdate[rootxsqone31 ~]# ntpdate time.windows.com[rootxsqone31 ~]# date
# 定时同步时间[rootxsqone31 ~]# crontab -e# 每5小时更新时间* */5 * * * /usr/sbin/ntpdate -u time.windows.com# 重新加载[rootxsqone31 ~]# service crond reload# 启动定时任务[rootxsqone31 ~]# service crond start2.7、设置/etc/hosts文件 添加映射 192.168.136.31 xsqone31
192.168.136.32 xsqone32
192.168.136.33 xsqone33
192.168.136.34 xsqone34
192.168.136.35 xsqone352.8、设置免密登录
# 配置免密登录
ssh-keygen -t rsa -P
# 将本地公钥拷贝到要免密登录的目标机器
ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 rootxsqone31
ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 rootxsqone32
ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 rootxsqone33
ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 rootxsqone34
ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 rootxsqone35
# 测试
ssh -p22 xsqone31
ssh -p22 xsqone32
ssh -p22 xsqone33
ssh -p22 xsqone34
ssh -p22 xsqone352.9、为后面可以主备替换安装psmisc
yum install psmisc -y三、安装JDK
3.1、安装jdk 将压缩包放入/opt/install目录下 创建/opt/soft目录使用以下脚本即可安装JDK 注意使用脚本需要赋可执行权限chmod 777 文件名 #! /bin/bash
echo auto install JDK beginging ...#global var
jdktrueif [ $jdk true ];thenecho jdk install set trueecho setup jdk 8tar -zxf /opt/install/jdk-8u321-linux-x64.tar.gz -C /opt/softmv /opt/soft/jdk1.8.0_321 /opt/soft/jdk180
# echo #jdk /etc/profile
# echo export JAVA_HOME/opt/soft/jdk180 /etc/profile
# echo export CLASS_PATH.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar /etc/profile
# echo export PATH$PATH:$JAVA_HOME/bin /etc/profilesed -i 54a\export JAVA_HOME/opt/soft/jdk180 /etc/profilesed -i 54a\export CLASS_PATH.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar /etc/profilesed -i 54a\export PATH$PATH:$JAVA_HOME/bin /etc/profilesed -i 54a\#jdk /etc/profile source /etc/profileecho setup jdk 8 successful!!!
fi3.2、测试jdk是否安装成功 如下若出现java版本即成功 [rootxsqone31 fileshell]# source /etc/profile
[rootxsqone31 fileshell]# java -version
java version 1.8.0_321
Java(TM) SE Runtime Environment (build 1.8.0_321-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.321-b07, mixed mode)
[rootxsqone31 fileshell]#
3.3、将配置文件和安装目录传输给其他主机
# 将安装目录传输给其他主机
scp -r /opt/soft/jdk180 rootxsqone32:/opt/soft/
scp -r /opt/soft/jdk180 rootxsqone33:/opt/soft/
scp -r /opt/soft/jdk180 rootxsqone34:/opt/soft/
scp -r /opt/soft/jdk180 rootxsqone35:/opt/soft/
# 将配置文件传输给其他主机
scp -r /etc/profile rootxsqone32:/etc/
scp -r /etc/profile rootxsqone33:/etc/
scp -r /etc/profile rootxsqone34:/etc/
scp -r /etc/profile rootxsqone35:/etc/四、搭建zookeeper集群
3.1、安装zookeeper 与java相似先安装一台再将安装目录与配置文件传输给其他主机 #! /bin/bash
echo auto install JDK beginging ...#global var
zktrue
hostnamehostnameif [ $zk true ];thenecho zookeeper install set trueecho setup zookeeper 345tar -zxf /opt/install/zookeeper-3.4.5-cdh5.14.2.tar.gz -C /opt/softmv /opt/soft/zookeeper-3.4.5-cdh5.14.2 /opt/soft/zk345cp /opt/soft/zk345/conf/zoo_sample.cfg /opt/soft/zk345/conf/zoo.cfgmkdir -p /opt/soft/zk345/datassed -i 12c dataDir/opt/soft/zk345/datas /opt/soft/zk345/conf/zoo.cfgecho 0 /opt/soft/zk345/datas/myidsed -i 54a\export ZOOKEEPER_HOME/opt/soft/zk345 /etc/profilesed -i 54a\export PATH$PATH:$ZOOKEEPER_HOME/bin /etc/profilesed -i 54a\#zookeeper /etc/profile
fi使用以上脚本后在/opt/soft/zk345/datas目录下修改zoo.cfg文件 vim /opt/soft/zk345/datas/zoo.cfg
# zookeeper集群的配置server.0xsqone31:2287:3387server.1xsqone32:2287:3387server.2xsqone33:2287:3387server.3xsqone34:2287:3387server.4xsqone35:2287:33873.2、将配置文件和安装目录传输给其他主机
# 将安装目录传输给其他主机
scp -r /opt/soft/zk345 rootxsqone32:/opt/soft/
scp -r /opt/soft/zk345 rootxsqone33:/opt/soft/
scp -r /opt/soft/zk345 rootxsqone34:/opt/soft/
scp -r /opt/soft/zk345 rootxsqone35:/opt/soft/
# 将配置文件传输给其他主机
scp -r /etc/profile rootxsqone32:/etc/
scp -r /etc/profile rootxsqone33:/etc/
scp -r /etc/profile rootxsqone34:/etc/
scp -r /etc/profile rootxsqone35:/etc/3.3、修改其他主机的id 根据zookeeper配置文件中的配置修改/opt/soft/zk345/datas/myid 3.4、集群使用脚本 以下脚本可以在一个主机启动集群中的全部zookeeper 启动./文件名 start 状态./文件名 status 关停./文件名 stop #! /bin/bashcase $1 in start){for i in xsqone31 xsqone32 xsqone33 xsqone34 xsqone35dossh $i source /etc/profile; /opt/soft/zk345/bin/zkServer.sh start done
};;
stop){for i in xsqone31 xsqone32 xsqone33 xsqone34 xsqone35dossh $i source /etc/profile; /opt/soft/zk345/bin/zkServer.sh stop done
};;
status){for i in xsqone31 xsqone32 xsqone33 xsqone34 xsqone35dossh $i source /etc/profile; /opt/soft/zk345/bin/zkServer.sh status done
};;
esac
3.5、测试zookeeper是否安装成功 使用3.4中启动和查看状态命令 如下图若有多台follower和一台leader即为成功。 四、搭建hadoop集群
4.1、安装hadoop
tar -zxvf /opt/install/hadoop-3.1.3.tar.gz -C ../soft/
mv /opt/soft/hadoop-3.1.3/ hadoop313
chown -R root:root /opt/soft/hadoop313/4.2-4.7文件目录为/opt/soft/hadoop313/etc/hadoop/ 4.2、core-site.xml
?xml version1.0 encodingUTF-8?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configurationpropertynamefs.defaultFS/namevaluehdfs://gky/valuedescription逻辑名称必须与hdfs-site.xml的dfs.nameservices保持一致/description/propertypropertynamehadoop.tmp.dir/namevalue/opt/soft/hadoop313/tmpdata/valuedescriptionnamenode上本地的临时文件夹/description/propertypropertynamehadoop.http.staticuser.user/namevalueroot/valuedescription默认用户/description/propertypropertynamehadoop.proxyuser.root.hosts/namevalue*/valuedescription/description/propertypropertynamehadoop.proxyuser.root.groups/namevalue*/valuedescription/description/propertypropertynameio.file.buffer.size/namevalue131072/valuedescription读写序列缓存为128KB/description/propertypropertynameha.zookeeper.quorum/namevaluexsqone31:2181,xsqone32:2181,xsqone33:2181,xsqone34:2181,xsqone35:2181/valuedescriptionZKFailoverController在自动故障转移中使用的ZooKeeper服务器地址列表以逗号分隔。/description/propertypropertynameha.zookeeper.session-timeout.ms/namevalue10000/valuedescriptionhadoop链接zookeeper的超时时长为10s/description/property
/configuration
4.3、hadoop-env.sh
export JAVA_HOME/opt/soft/jdk180
export HDFS_NAMENODE_USERroot
export HDFS_DATANODE_USERroot
export HDFS_SECONDARYNAMENODE_USERroot
export HDFS_JOURNALNODE_USERroot
export HDFS_ZKFC_USERroot
export YARN_RESOURCEMANAGER_USERroot
export YARN_NODEMANAGER_USERroot4.4、hdfs-site.xml
?xml version1.0 encodingUTF-8?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configurationpropertynamedfs.replication/namevalue3/valuedescriptionhadoop中每一个block的备份数/description/propertypropertynamedfs.namenode.name.dir/namevalue/opt/soft/hadoop313/data/dfs/name/valuedescriptionnamenode上存储hdfs名字的空间元数据路径/description/propertypropertynamedfs.datanode.data.dir/namevalue/opt/soft/hadoop313/data/dfs/data/valuedescriptiondatanode上数据块的物理存储路径/description/propertypropertynamedfs.namenode.secondary.http-address/namevaluexsqone31:9869/valuedescription/description/propertypropertynamedfs.nameservices/namevaluegky/valuedescription指定hdfs的nameservice需要和core-site.xml中保持一致/description/propertypropertynamedfs.ha.namenodes.gky/namevaluenn1,nn2,nn3/valuedescriptiongky为集群的逻辑名称映射两个namenode逻辑名/description/propertypropertynamedfs.namenode.rpc-address.gky.nn1/namevaluexsqone31:9000/valuedescriptionnn1的rpc通信地址/description/propertypropertynamedfs.namenode.http-address.gky.nn1/namevaluexsqone31:9870/valuedescriptionnn1的http通信地址/description/propertypropertynamedfs.namenode.rpc-address.gky.nn2/namevaluexsqone32:9000/valuedescriptionnn2的rpc通信地址/description/propertypropertynamedfs.namenode.http-address.gky.nn2/namevaluexsqone32:9870/valuedescriptionnn2的http通信地址/description/propertypropertynamedfs.namenode.rpc-address.gky.nn3/namevaluexsqone33:9000/valuedescriptionnn3的rpc通信地址/description/propertypropertynamedfs.namenode.http-address.gky.nn3/namevaluexsqone33:9870/valuedescriptionnn3的http通信地址/description/propertypropertynamedfs.namenode.shared.edits.dir/namevalueqjournal://xsqone31:8485;xsqone32:8485;xsqone33:8485/gky/valuedescription指定NameNode的edits元数据的共享存储位置journal列表/description/propertypropertynamedfs.journalnode.edits.dir/namevalue/opt/soft/hadoop313/data/journaldata/valuedescription指定JournalNode在本地磁盘存放数据的位置/description/property!-- 容错--propertynamedfs.ha.automatic-failover.enabled/namevaluetrue/valuedescription开启NameNode故障自动切换/description/propertypropertynamedfs.client.failover.proxy.provider.gky/namevalueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/valuedescription失败后自动切换的实现方式/description/propertypropertynamedfs.ha.fencing.methods/namevaluesshfence/valuedescription防止脑裂的处理/description/propertypropertynamedfs.ha.fencing.ssh.private-key-files/namevalue/root/.ssh/id_rsa/valuedescription使用sshfence隔离机制时需要使用到免密登录/description/propertypropertynamedfs.permissions.enabled/namevaluefalse/valuedescription关闭HDFS权限验证/description/propertypropertynamedfs.image.transfer.bandwidthPerSec/namevalue1048576/valuedescription/description/propertypropertynamedfs.block.scanner.volume.bytes.per.second/namevalue1048576/valuedescription/description/property
/configuration
4.5、mapred-site.xml
?xml version1.0?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configurationpropertynamemapreduce.framework.name/namevalueyarn/valuedescriptionjob执行框架local,classic or yarn/descriptionfinaltrue/final/propertypropertynamemapreduce.application.classpath/namevalue/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*/valuedescriptionjob执行框架local,classic or yarn/description/propertypropertynamemapreduce.jobhistory.address/namevaluexsqone31:10020/value/propertypropertynamemapreduce.jobhistory.webapp.address/namevaluexsqone31:19888/value/propertypropertynamemapreduce.map.memory.mb/namevalue1024/valuedescriptionmap阶段的task工作内存/description/propertypropertynamemapreduce.reduce.memory.mb/namevalue2048/valuedescriptionreduce阶段的task工作内存/description/property
/configuration
4.6、yarn-site.xml
?xml version1.0?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--
configurationpropertynameyarn.resourcemanager.ha.enabled/namevaluetrue/valuedescription开启resourcemanager高可用/description/propertypropertynameyarn.resourcemanager.cluster-id/namevalueyrcabc/valuedescription指定yarn集群中的id/description/propertypropertynameyarn.resourcemanager.ha.rm-ids/namevaluerm1,rm2,rm3/valuedescription指定resourcemanager的名字/description/propertypropertynameyarn.resourcemanager.hostname.rm1/namevaluexsqone33/valuedescription设置rm1的名字/description/propertypropertynameyarn.resourcemanager.hostname.rm2/namevaluexsqone34/valuedescription设置rm2的名字/description/propertypropertynameyarn.resourcemanager.hostname.rm3/namevaluexsqone35/valuedescription设置rm3的名字/description/propertypropertynameyarn.resourcemanager.webapp.address.rm1/namevaluexsqone33:8088/valuedescription/description/propertypropertynameyarn.resourcemanager.webapp.address.rm2/namevaluexsqone34:8088/valuedescription/description/propertypropertynameyarn.resourcemanager.webapp.address.rm3/namevaluexsqone35:8088/valuedescription/description/propertypropertynameyarn.resourcemanager.zk-address/namevaluexsqone31:2181,xsqone32:2181,xsqone33:2181,xsqone34:2181,xsqone35:2181/valuedescription指定zk集群地址/description/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/valuedescription运行mapreduce程序的时候必须配置的附属服务/description/propertypropertynameyarn.nodemanager.local-dirs/namevalue/opt/soft/hadoop313/tmpdata/yarn/local/valuedescriptionnodemanager本地存储目录/description/propertypropertynameyarn.nodemanager.log-dirs/namevalue/opt/soft/hadoop313/tmpdata/yarn/log/valuedescriptionnodemanager本地日志目录/description/propertypropertynameyarn.nodemanager.resource.memory-mb/namevalue2048/valuedescriptionresource进程的工作内存/description/propertypropertynameyarn.nodemanager.resource.cpu-vcores/namevalue2/valuedescriptionresource工作中所能使用机器的内核数/description/propertypropertynameyarn.scheduler.minimum-allocation-mb/namevalue256/valuedescription每个容器请求的最小分配/description/propertypropertynameyarn.log-aggregation-enable/namevaluetrue/valuedescription开启日志聚合/description/propertypropertynameyarn.log-aggregation.retain-seconds/namevalue86400/valuedescription日志保留多少秒/description/propertypropertynameyarn.nodemanager.vmem-check-enabled/namevaluefalse/valuedescription/description/propertypropertynameyarn.application.classpath/namevalue/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*/valuedescription/description/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME/valuedescription/description/property/configuration
4.7、workers
xsqone31
xsqone32
xsqone33
xsqone34
xsqone354.8、/etc/profile
#hadoop
export JAVA_LIBRARY_PATH/opt/soft/hadoop313/lib/native
export HADOOP_HOME/opt/soft/hadoop313
export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib4.9、将配置文件和安装目录传输给其他主机
# 将安装目录传输给其他主机
scp -r /opt/soft/hadoop313 rootxsqone32:/opt/soft/
scp -r /opt/soft/hadoop313 rootxsqone33:/opt/soft/
scp -r /opt/soft/hadoop313 rootxsqone34:/opt/soft/
scp -r /opt/soft/hadoop313 rootxsqone35:/opt/soft/
# 将配置文件传输给其他主机
scp -r /etc/profile rootxsqone32:/etc/
scp -r /etc/profile rootxsqone33:/etc/
scp -r /etc/profile rootxsqone34:/etc/
scp -r /etc/profile rootxsqone35:/etc/五、集群首次启动
集群首次启动
1、启动zookeeper集群
2、在xsqone31-33 启动journalnodehdfs --daemon start journalnode
3、在xsqone31格式化hdfs hdfs namenode -format
4、在xsqone31启动namenodehdfs --daemon start namenode
5、在32-33机器上同步namenode信息hdfs namenode -bootstrapStandby
6、在32-33机器上启动namenodehdfs --daemon start namenode查看namenode节点状态hdfs haadmin -getServiceState nn1|nn2|nn3
7、关闭所有dfs的服务stop-dfs.sh
8、格式化ZKhdfs zkfc -formatZK
9、启动hdfsstart-dfs.sh10、启动yarnstart-yarn.sh查看resourcemanager节点状态yarn rmadmin -getServiceState rm1|rm2|rm3 六、测试集群是否成功
6.1、上传文件测试
此处有上传命令
6.2使用jar包运行测试
jar包目录为 /opt/soft/hadoop313/share/hadoop/mapreduce
