上海 网站建设平台 补贴,电子商务网站建设教材,上海专业seo公司,怎么制作自己的二维码一、系统文件配置集群部署规划NameNode和SecondaryNameNode不要安装在同一台服务器ResourceManager也很消耗内存#xff0c;不要和NameNode、SecondaryNameNode放在同一台机器上。这里装了四台机器#xff0c;ant151,ant152,ant153,ant154。ant151ant152ant153ant154NameNode…一、系统文件配置集群部署规划NameNode和SecondaryNameNode不要安装在同一台服务器ResourceManager也很消耗内存不要和NameNode、SecondaryNameNode放在同一台机器上。这里装了四台机器ant151,ant152,ant153,ant154。ant151ant152ant153ant154NameNodeNameNodeDataNodeDataNodeDataNodeDataNodeNodeManagerNodeManagerNodeManagerNodeManagerResourceManagerResourceManagerJournalNodeJournalNodeJournalNodeDFSZKFControllerDFSZKFControllerzk0zk1zk2配置文件说明Hadoop配置文件分为默认配置文件和自定义配置文件只有用户想修改某一默认配置值时才需要修改自定义配置文件。core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml四个配置文件放在$HADOOP_HOME/etc/hadoop路径下。3.配置集群core-site.xml propertynamefs.defaultFS/namevaluehdfs://gky/valuedescription逻辑名称必须与hdfs-site.xml中的dfs.nameservices值保持一致/description/propertypropertynamehadoop.tmp.dir/namevalue/opt/soft/hadoop313/tmpdata/valuedescriptionnamenode上本地的hadoop临时文件夹/description/propertypropertynamehadoop.http.staticuser.user/namevalueroot/valuedescription默认用户/description/propertypropertynamehadoop.proxyuser.root.hosts/namevalue*/valuedescription/description/propertypropertynamehadoop.proxyuser.root.groups/namevalue*/valuedescription/description/propertypropertynameio.file.buffer.size/namevalue131072/valuedescription读写文件的buffer大小为128K/description/propertypropertynameha.zookeeper.quorum/namevalueant151:2181,ant152:2181,ant153:2181/valuedescription/description/propertypropertynameha.zookeeper.session-timeout.ms/namevalue10000/valuedescriptionhadoop链接zookeeper的超时时长设置为10s/description/propertyhdfs-site.xml propertynamedfs.replication/namevalue3/valuedescriptionHadoop中每一个block的备份数/description/propertypropertynamedfs.namenode.name.dir/namevalue/opt/soft/hadoop313/data/dfs/name/valuedescriptionnamenode上存储hdfs名字空间元数据目录/description/propertypropertynamedfs.datanode.data.dir/namevalue/opt/soft/hadoop313/data/dfs/data/valuedescriptiondatanode上数据块的物理存储位置/description/propertypropertynamedfs.namenode.secondary.http-address/namevalueant151:9869/valuedescription/description/propertypropertynamedfs.nameservices/namevaluegky/valuedescription指定hdfs的nameservice,需要和core-site.xml中保持一致/description/propertypropertynamedfs.ha.namenodes.gky/namevaluenn1,nn2/valuedescriptiongky为集群的逻辑名称映射两个namenode逻辑名/description/propertypropertynamedfs.namenode.rpc-address.gky.nn1/namevalueant151:9000/valuedescriptionnamenode1的RPC通信地址/description/propertypropertynamedfs.namenode.http-address.gky.nn1/namevalueant151:9870/valuedescriptionnamenode1的http通信地址/description/propertypropertynamedfs.namenode.rpc-address.gky.nn2/namevalueant152:9000/valuedescriptionnamenode2的RPC通信地址/description/propertypropertynamedfs.namenode.http-address.gky.nn2/namevalueant152:9870/valuedescriptionnamenode2的http通信地址/description/propertypropertynamedfs.namenode.shared.edits.dir/namevalueqjournal://ant151:8485;ant152:8485;ant153:8485/gky/valuedescription指定NameNode的edits元数据的共享存储位置(JournalNode列表)/description/propertypropertynamedfs.journalnode.edits.dir/namevalue/opt/soft/hadoop313/data/journaldata/valuedescription指定JournalNode在本地磁盘存放数据的位置/description/property !-- 容错 --propertynamedfs.ha.automatic-failover.enabled/namevaluetrue/valuedescription开启NameNode故障自动切换/description/propertypropertynamedfs.client.failover.proxy.provider.gky/namevalueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/valuedescription失败后自动切换的实现方式/description/propertypropertynamedfs.ha.fencing.methods/namevaluesshfence/valuedescription防止脑裂的处理/description/propertypropertynamedfs.ha.fencing.ssh.private-key-files/namevalue/root/.ssh/id_rsa/valuedescription使用sshfence隔离机制时需要ssh免密登陆/description/property propertynamedfs.permissions.enabled/namevaluefalse/valuedescription关闭HDFS操作权限验证/description/propertypropertynamedfs.image.transfer.bandwidthPerSec/namevalue1048576/valuedescription/description/property propertynamedfs.block.scanner.volume.bytes.per.second/namevalue1048576/valuedescription/description/propertymapred-site.xml propertynamemapreduce.framework.name/namevalueyarn/valuedescriptionjob执行框架 local, classic or yarn/descriptionfinaltrue/final/propertypropertynamemapreduce.application.classpath/namevalue/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*/value/propertypropertynamemapreduce.jobhistory.address/namevalueant151:10020/value/propertypropertynamemapreduce.jobhistory.webapp.address/namevalueant151:19888/value/propertypropertynamemapreduce.map.memory.mb/namevalue1024/valuedescriptionmap阶段的task工作内存/description/propertypropertynamemapreduce.reduce.memory.mb/namevalue2048/valuedescriptionreduce阶段的task工作内存/description/propertyyarn-site.xml propertynameyarn.resourcemanager.ha.enabled/namevaluetrue/valuedescription开启resourcemanager高可用/description/propertypropertynameyarn.resourcemanager.cluster-id/namevalueyrcabc/valuedescription指定yarn集群中的id/description/propertypropertynameyarn.resourcemanager.ha.rm-ids/namevaluerm1,rm2/valuedescription指定resourcemanager的名字/description/propertypropertynameyarn.resourcemanager.hostname.rm1/namevalueant153/valuedescription设置rm1的名字/description/propertypropertynameyarn.resourcemanager.hostname.rm2/namevalueant154/valuedescription设置rm2的名字/description/propertypropertynameyarn.resourcemanager.webapp.address.rm1/namevalueant153:8088/valuedescription/description/propertypropertynameyarn.resourcemanager.webapp.address.rm2/namevalueant154:8088/valuedescription/description/property propertynameyarn.resourcemanager.zk-address/namevalueant151:2181,ant152:2181,ant153:2181/valuedescription指定zk集群地址/description/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/valuedescription运行mapreduce程序必须配置的附属服务/description/propertypropertynameyarn.nodemanager.local-dirs/namevalue/opt/soft/hadoop313/tmpdata/yarn/local/valuedescriptionnodemanager本地存储目录/description/propertypropertynameyarn.nodemanager.log-dirs/namevalue/opt/soft/hadoop313/tmpdata/yarn/log/valuedescriptionnodemanager本地日志目录/description/propertypropertynameyarn.nodemanager.resource.memory-mb/namevalue2048/valuedescriptionresource进程的工作内存/description/propertypropertynameyarn.nodemanager.resource.cpu-vcores/namevalue2/valuedescriptionresource工作中所能使用机器的内核数/description/propertypropertynameyarn.scheduler.minimum-allocation-mb/namevalue256/valuedescription/description/propertypropertynameyarn.log-aggregation-enable/namevaluetrue/valuedescription/description/propertypropertynameyarn.log-aggregation.retain-seconds/namevalue86400/valuedescription日志保留多少秒/description/propertypropertynameyarn.nodemanager.vmem-check-enabled/namevaluefalse/valuedescription/description/propertypropertynameyarn.application.classpath/namevalue/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*/valuedescription/description/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME/valuedescription/description/propertyhadoop-env.shexport JAVA_HOME/opt/soft/jdk180
export HDFS_NAMENODE_USERroot
export HDFS_DATANODE_USERroot
export HDFS_SECONDARYNAMENODE_USERroot
export HDFS_JOURNALNODE_USERroot
export HDFS_ZKFC_USERroot
export YARN_RESOURCEMANAGER_USERroot
export YARN_NODEMANAGER_USERrootworkersant151
ant152
ant153
ant154二、集群首次启动启动zk集群可以直接运行脚本文件代码[rootant151 shell]# ./zkop.sh start 启动ant151,ant152,ant153的journalnode服务[rootant151 shell]# hdfs --daemon start journalnode在ant151格式化hfds namenode: [rootant151 shell]# hdfs namenode -format在ant151启动namenode服务hdfs --daemon start namenode[rootant151 shell]# hdfs --daemon start namenode在ant152机器上同步namenode信息[rootant151 shell]# hdfs namenode -bootstrapStandby在ant152启动namenode服务hdfs --daemon start namenode[rootant152 soft]# hdfs --daemon start namenode 查看namenode节点状态hdfs haadmin -getServiceState nn1|nn2[rootant152 soft]# hdfs haadmin -getServiceState nn1关闭所有dfs有关的服务[rootant151 soft]# stop-dfs.sh格式化zk[rootant151 soft]# hdfs zkfc -formatZK启动dfs[rootant151 soft]# start-dfs.sh启动yarn: [rootant151 soft]# start-yarn.sh [rootant151 soft]# start-yarn.sh 查看resourcemanager节点状态[rootant151 soft]# yarn rmadmin -getServiceState rm1rm1状态standbyrm2状态active当前进程状态kill掉active进程尝试访问无法链接恢复ant152的namenode进程
