微网站如何建设,江门建站网站模板,佛山建设小学网站,网站建设分金手指专业二五金翅大鹏盖世英#xff0c;展翅金鹏盖世雄。
穿云燕子锡今鸽#xff0c;踏雪无痕花云平。
---------------- 本文密钥#xff1a;338 -----------------
本文描述了在macbook pro的macos上安装hadoop3的过程#xff0c;也可以作为在任何类linux平台上安装hadoop3借鉴。 …金翅大鹏盖世英展翅金鹏盖世雄。
穿云燕子锡今鸽踏雪无痕花云平。
---------------- 本文密钥338 -----------------
本文描述了在macbook pro的macos上安装hadoop3的过程也可以作为在任何类linux平台上安装hadoop3借鉴。
1、准备工作
确保已安装jdk本地已安装jdk1.8.0_221。
export JAVA_HOME/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home/
下载Hadoop-3.0.0
https://archive.apache.org/dist/hadoop/common/hadoop-3.0.0/hadoop-3.0.0.tar.gz
放到安装目录我使用的是/user/local/
解压tar zxf hadoop-3.0.0.tar.gz
方便后续维护建立符号链接ln -s hadoop-3.0.0 hadoop 2、在~/.bashrc文件中配置HADOOP_HOME
export HADOOP_HOME/user/local/hadoop
export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_HOME$HADOOP_HOME
3、配置本地ssh免密登录
ssh localhost # 若成功则已配置跳过cd ~/.ssh
# 生成秘钥文件
ssh-keygen -t rsa -P -f ~/.ssh/id_rsa# 将公钥文件加入授权文件中
cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys#测试一下
ssh localhost
4、修改core-site.xml
confiqurationpropertynamehadoop.tmp.dir/namevalue/user/local/hadoop/tmp/valuedescriptionA base for other temporary directoriesdescription/propertypropertynamefs.default.name/namevaluehdfs://localhost:9000/value/property
/configuration
5、修改mapred-site.xml
configurationpropertynamemapredjob.tracker/namevaluelocalhost:9010/value/propertypropertynamemapreduce.framework.name/namevalueyarn/value/property
/configuration
6、修改hdfs-site.xml
configuration!--伪分布式配置--propertynamedfs.replication/namevalue1/value/property
/configuration
7、修改yarn-site.xml
configurationpropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/property!--集群配置--!--propertynameyarn.resourcemanager.hostname/namevaluemaster/value/property--
/configuration
8、修改hadoop-env.sh
export JAVA_HOME/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home/
export HADOOP_OPTS$HADOOP_OPTS -Djava.net.preferIPv4Stacktrue -Djava.security.krb5.realm -Djava.security.krb5.kdc
export HADOOP_OS_TYPE${HADOOP_OS_TYPE:-$(uname -s)}
# Under certain conditions, Java on OS X will throw SCDynamicStore errors
# in the system logs.
# See HADOOP-8719 for moreinformation. If one needs Kerberos
# support on 0S Xone will want to change/remove this extra bit
case ${HADOOP_OS_TYPE} inDarwin*)export HADOOP_OPTS${HADOOP_OPTS} -Djava.security.krb5.realm export HADOOP_OPTS${HADOOP_OPTS} -Djava.security.krb5.kdc export HADOOP_OPTS${HADOOP_OPTS} -Djava.security.krb5.conf ;;
esac
9、启动
cd $HADOOP_HOME
# 格式化namenode
./bin/hdfs namenode -format./sbin/start-all.sh
成功 检查jps 检查管理页面
Hadoop管理页面http://localhost:8088/cluster
NameNodehttp://localhost:9870/
DataNodehttp://localhost:9864/
HDFS目录http://localhost:9870/explorer.html#/ 相关问题
问题1启动时start-all.sh出现错误
错误WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
解决方法
需要下载hadoop源码包hadoop-3.0.0-src.tar.gz重新编译后将安全库拷贝到hadoop目录中
tar zxf hadoop-3.0.0-src.tar.gz
cd hadoop-3.0.0-src
mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skiptrue# 成功后
cp ./hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.* ../hadoop-3.0.0/lib/native/
cd ../hadoop-3.0.0
问题2启动时namenode错误
WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /user/local/hadoop/tmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible
解决方法
执行一下格式化namenode命令
./bin/hdfs namenode -format 其他错误如目录权限等问题具体情况具体分析。
