如何做全景素材网站,如何在外管局网站上做延期,钻石网站建设,建设小说网站用什么软件Day01-ElasticSearch的单点部署#xff0c;集群部署#xff0c;多实例部署#xff0c;es-head和postman环境搭建 0、ElasticSearch的简单介绍1、ElasticSearch的单点部署2、ElasticSearch的集群部署3、基于二进制部署ElasticSearch3.1 准备阶段3.2 部署阶段3.3 使用systemct… Day01-ElasticSearch的单点部署集群部署多实例部署es-head和postman环境搭建 0、ElasticSearch的简单介绍1、ElasticSearch的单点部署2、ElasticSearch的集群部署3、基于二进制部署ElasticSearch3.1 准备阶段3.2 部署阶段3.3 使用systemctl管理ES服务3.4 使用oracle JDK管理ES服务3.5 修改ES环境的堆(heap)内存大小3.6 ES集群的多实例部署3.7 常见报错 0、ElasticSearch的简单介绍
Elasticstack
Elastic$earch 日志存储和检索 filebeat 用于日志收集 logstash 日志转换也可以用于日志采集 kibana 用于日志展示 TB(PB)级别日志处理 kafkazookeeper
ElasticSearch: 单点部署 集群部署二 ES集群的常见术语 分片副本索引文档… 集群的API基础操作 索引管理文档管理 集群环境准备
IP地址配置主机名10.0.0.101CPU:2C MEMORY:4G DISK:50Gelk101.oldboyedu.com10.0.0.102CPU:2C MEMORY:4G DISK:50Gelk102.oldboyedu.com10.0.0.103CPU:2C MEMORY:4G DISK:50Gelk103.oldboyedu.com
配置好后拍快照
1、ElasticSearch的单点部署
(1)下载ES软件包 https://www.elastic.co/cn/downloads
(2)老男孩线下同学下载
curl -o elasticsearch-7.17.5-x86_64.rpm http://192.168.15.253/ElasticStack/softwares/rpm/elasticsearch-7.17.5-x86_64.rpm(3)安装es
rpm -ivh elasticsearch-7.17.5-x86_64.rpm (4)修改es的配置文件
vim /etc/elasticsearch/elasticsearch.yml
...
# ES服务监听对外暴露服务的地址
network.host: 0.0.0.0# 指定ES集群的节点IP
discovery.seed_hosts: [10.0.0.101]# 指定参与master选举的节点
cluster.initial_master_nodes: [10.0.0.101](5)启动ES服务
systemctl enable --now elasticsearch(6)验证节点是否正常工作
[rootelk101 ~]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
...
LISTEN 0 16384 [::]:9200 [::]:*
LISTEN 0 16384 [::]:9300 [::]:* 面试题1: 9200端口作用: 对ES集群外部提供http/https服务。可以理解为对客户端提供服务。 9300端口作用: 对ES集群内部进行数据通信传输端口。走的时候tcp协议。
(7)客户端验证
curl 10.0.0.101:9200如果遇到集群的uuid为_na_情况时可以执行如下操作:
# 两块网卡时打开指定参与master选举的节点
vim /etc/elasticsearch/elasticsearch.yml
cluster.initial_master_nodes: [10.0.0.101]
# 然后执行
systemctl stop elasticsearch.service
rm -rf /var/lib/elasticsearch/* /var/log/elasticsearch/* /tmp/*
systemctl start elasticsearch.service
curl 10.0.0.101:92002、ElasticSearch的集群部署
(1)下载ES软件包 https://www.elastic.co/cn/downloads
(2)老男孩线下同学下载
curl -o elasticsearch-7.17.5-x86_64.rpm http://192.168.15.253/ElasticStack/softwares/rpm/elasticsearch-7.17.5-x86_64.rpm(3)所有节点安装es
rpm -ivh elasticsearch-7.17.5-x86_64.rpm (4)elk101.oldboyedu.com做数据清空
systemctl stop elasticsearch.service
rm -rf /var/lib/elasticsearch/* /var/log/elasticsearch/* /tmp/* (5)修改es的配置文件
vim /etc/elasticsearch/elasticsearch.yml
...# 指定ES集群的名称
cluster.name: oldboyedu-linux85# ES服务监听对外暴露服务的地址
network.host: 0.0.0.0# 指定ES集群的节点IP
discovery.seed_hosts: [10.0.0.101,10.0.0.102,10.0.0.103]# 指定参与master选举的节点
cluster.initial_master_nodes: [10.0.0.101,10.0.0.102,10.0.0.103](6)将配置文件分发到其他两个节点
scp /etc/elasticsearch/elasticsearch.yml 10.0.0.102:/etc/elasticsearch/elasticsearch.yml
scp /etc/elasticsearch/elasticsearch.yml 10.0.0.103:/etc/elasticsearch/elasticsearch.yml(6)所有节点启动ES服务
systemctl enable --now elasticsearch(7)验证ES集群节点是否正常工作
[rootelk101.oldboyedu.com ~]# curl 10.0.0.101:9200/_cat/nodes # *在谁那谁就是老大
10.0.0.103 11 71 51 1.28 0.38 0.13 cdfhilmrstw - elk103.oldboyedu.com
10.0.0.102 15 73 46 1.10 0.34 0.12 cdfhilmrstw - elk102.oldboyedu.com
10.0.0.101 6 92 1 0.07 0.11 0.12 cdfhilmrstw * elk101.oldboyedu.com# 半数以上机器存活则集群正常为了防止脑裂半数以上机器不存活的话集群不提供对外服务[rootelk101.oldboyedu.com ~]#
[rootelk101.oldboyedu.com ~]# curl 10.0.0.101:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15 m node.role master name
10.0.0.103 12 71 0 0.72 0.34 0.13 cdfhilmrstw - elk103.oldboyedu.com
10.0.0.102 16 73 0 0.61 0.31 0.12 cdfhilmrstw - elk102.oldboyedu.com
10.0.0.101 7 92 0 0.04 0.10 0.11 cdfhilmrstw * elk101.oldboyedu.comheap.percent # 堆内存使用百分比
ram.percent # 内存百分比3、基于二进制部署ElasticSearch
3.1 准备阶段
(1)elk101配置集群免密登录及同步脚本 1.1 修改主机列表
cat /etc/hosts EOF
10.0.0.101 elk101.oldboyedu.com
10.0.0.102 elk102.oldboyedu.com
10.0.0.103 elk103.oldboyedu.com
EOF1.2 elk101节点上生成密钥对
ssh-keygen -t rsa -P -f ~/.ssh/id_rsa -q1.3 elk101配置所有集群节点的免密登录
for ((host_id101;host_id103;host_id));do ssh-copy-id elk${host_id}.oldboyedu.com ;done1.4 链接测试
ssh elk101.oldboyedu.com
ssh elk102.oldboyedu.com
ssh elk103.oldboyedu.com1.5 所有节点安装rsync数据同步工具
yum -y install rsync1.6 编写同步脚本
cat /usr/local/sbin/data_rsync.sh EOF
#!/bin/bash# Auther: Jason Yinif [ $# -ne 1 ];thenecho Usage: $0 /path/to/file(绝对路径)exit
fi # 判断文件是否存在if [ ! -e $1 ];thenecho [ $1 ] dir or file not find!exit
fi# 获取父路径fullpathdirname $1# 获取子路径basenamebasename $1# 进入到父路径cd $fullpathfor ((host_id102;host_id103;host_id))do# 使得终端输出变为绿色tput setaf 2echo rsyncing elk${host_id}.oldboyedu.com: $basename # 使得终端恢复原来的颜色tput setaf 7# 将数据同步到其他两个节点rsync -apz $basename whoamielk${host_id}.oldboyedu.com:$fullpathif [ $? -eq 0 ];thenecho 命令执行成功!fi
done
EOF1.7 给脚本授权
chmod x /usr/local/sbin/data_rsync.sh3.2 部署阶段
(2)elk101下载ES软件包
curl -o elasticsearch-7.17.5-linux-x86_64.tar.gz http://192.168.15.253/ElasticStack/softwares/binary/elasticsearch-7.17.5-linux-x86_64.tar.gz(3)所有节点创建运行ES服务的用户
useradd -u 2023 oldboyedu(4)elk101创建ElasticSearch的工作目录 4.1 可以如下操作
mkdir -pv /oldboyedu/{data,logs,softwares}/es7
chown oldboyedu:oldboyedu -R /oldboyedu/{softwares,data,logs}/es7/
ll /oldboyedu/{data,logs,softwares}/es7 -d4.2 也可以如下操作推荐
install -d /oldboyedu/{data,logs,softwares}/es7 -o oldboyedu -g oldboyedu(5)elk101解压软件包
tar xf elasticsearch-7.17.5-linux-x86_64.tar.gz -C /oldboyedu/softwares/es7/(6)elk101修改配置文件
[rootelk101.oldboyedu.com ~]# egrep -v ^#|^$ /oldboyedu/softwares/es7/elasticsearch-7.17.5/config/elasticsearch.yml
cluster.name: oldboyedu-linux85-binary
path.data: /oldboyedu/data/es7
path.logs: /oldboyedu/logs/es7
network.host: 0.0.0.0
discovery.seed_hosts: [elk101.oldboyedu.com,elk102.oldboyedu.com,elk103.oldboyedu.com]
cluster.initial_master_nodes: [elk101.oldboyedu.com,elk102.oldboyedu.com,elk103.oldboyedu.com](7)elk101同步程序目录
[rootelk101.oldboyedu.com ~]# data_rsync.sh /oldboyedu(8)elk101节点针对ES基础调优 8.1 修改文件打开数量上线修改后需要断开会话
[rootelk101.oldboyedu.com ~]# cat /etc/security/limits.d/es7.conf* soft nofile 65535
* hard nofile 131070
* hard nproc 8192[rootelk101.oldboyedu.com ~]# ulimit -Sn
65535[rootelk101.oldboyedu.com ~]# ulimit -Hn
1310708.2 调大内核虚拟内存映射值
[rootelk101.oldboyedu.com ~]# cat /etc/sysctl.d/es.conf
vm.max_map_count524288[rootelk101.oldboyedu.com ~]# sysctl -q vm.max_map_count
vm.max_map_count 65530[rootelk101.oldboyedu.com ~]# sysctl -f /etc/sysctl.d/es.conf # 生效
vm.max_map_count 524288[rootelk101.oldboyedu.com ~]# sysctl -q vm.max_map_count
vm.max_map_count 5242888.3 同步调优文件
[rootelk101.oldboyedu.com ~]# data_rsync.sh /etc/sysctl.d/es.conf
[rootelk101.oldboyedu.com ~]# data_rsync.sh /etc/security/limits.d/es7.conf
[rootelk101.oldboyedu.com ~]# data_rsync.sh /etc/hosts(9)所有节点启动服务
9.1 先断开连接
让文件打开数量上限生效。
9.2 使得内核参数生效
sysctl -f /etc/sysctl.d/es.conf 9.3 后台启动es服务
chown oldboyedu:oldboyedu -R /oldboyedu/{softwares,data,logs}/es7/
su oldboyedu -c /oldboyedu/softwares/es7/elasticsearch-7.17.5/bin/elasticsearch -d (10)验证服务是否正常
[rootelk101.oldboyedu.com ~]# curl 10.0.0.101:9200
{name : elk101.oldboyedu.com,cluster_name : oldboyedu-linux85-binary,cluster_uuid : l0Vl0fYMSmSun_Mnv3dH7g,version : {number : 7.17.5,build_flavor : default,build_type : tar,build_hash : 8d61b4f7ddf931f219e3745f295ed2bbc50c8e84,build_date : 2022-06-23T21:57:28.736740635Z,build_snapshot : false,lucene_version : 8.11.1,minimum_wire_compatibility_version : 6.8.0,minimum_index_compatibility_version : 6.0.0-beta1},tagline : You Know, for Search
}[rootelk101.oldboyedu.com ~]# curl 10.0.0.101:9200/_cat/nodes
10.0.0.101 27 96 38 0.68 0.26 0.14 cdfhilmrstw * elk101.oldboyedu.com
10.0.0.102 23 88 38 0.49 0.21 0.14 cdfhilmrstw - elk102.oldboyedu.com
10.0.0.103 21 88 42 1.25 0.42 0.22 cdfhilmrstw - elk103.oldboyedu.com补充若集群UUID出现_na_状态时做如下动作所有节点操作
pkill java
rm -rf /oldboyedu/data/es7/* /oldboyedu/logs/es7/* /tmp/*3.3 使用systemctl管理ES服务
1.elk101停止ES服务
pkill java2.elk101编写ES启动脚本
cat /usr/lib/systemd/system/es7.service EOF
[Unit]
Descriptionoldboyedu linux85 es7
Afternetwork.target[Service]
Typesimple
ExecStart/oldboyedu/softwares/es7/elasticsearch-7.17.5/bin/elasticsearch
Useroldboyedu
LimitNOFILE131070[Install]
WantedBymulti-user.target
EOF3.elk101加载脚本
systemctl daemon-reload4.elk101设置开机自启动
systemctl enable --now es75.elk101查看服务是否正常运行
systemctl status es7
ss -ntl
curl 10.0.0.101:9200/_cat/nodes6.其他节点使用脚本
data_rsync.sh /usr/lib/systemd/system/es7.service7.其他节点使用脚本管理
pkill java
systemctl enable --now es7
systemctl status es73.4 使用oracle JDK管理ES服务
(1)elk101下载JDK环境
curl -o jdk-8u291-linux-x64.tar.gz http://192.168.15.253/ElasticStack/softwares/binary/jdk-8u291-linux-x64.tar.gz(2)elk101解压JDK软件包
tar xf jdk-8u291-linux-x64.tar.gz -C /oldboyedu/softwares/(3)elk101配置系统环境变量
[rootelk101.oldboyedu.com ~]# cat /etc/profile.d/jdk.sh
#!/bin/bashexport JAVA_HOME/oldboyedu/softwares/jdk1.8.0_291
export PATH$PATH:$JAVA_HOME/bin[rootelk101 ~]# source /etc/profile.d/jdk.sh
[rootelk101 ~]# which java
/oldboyedu/softwares/jdk1.8.0_291/bin/java
[rootelk101.oldboyedu.com ~]# java -version
java version 1.8.0_291
Java(TM) SE Runtime Environment (build 1.8.0_291-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.291-b10, mixed mode)(4)elk101修改启动脚本
cat /usr/lib/systemd/system/es7.service EOF
[Unit]
Descriptionoldboyedu linux85 es7
Afternetwork.target[Service]
Typesimple
EnvironmentJAVA_HOME/oldboyedu/softwares/jdk1.8.0_291
ExecStart/oldboyedu/softwares/es7/elasticsearch-7.17.5/bin/elasticsearch
Useroldboyedu
LimitNOFILE131070
LimitNPROC8192[Install]
WantedBymulti-user.target
EOFsystemctl daemon-reload
systemctl start es7(5)elk101重启服务
systemctl restart es7(6)elk101查看服务是否正常
[rootelk101.oldboyedu.com ~]# curl 10.0.0.101:9200/_cat/nodes
10.0.0.101 17 74 6 0.17 0.28 0.19 cdfhilmrstw - elk101.oldboyedu.com
10.0.0.102 11 90 0 0.03 0.03 0.05 cdfhilmrstw * elk102.oldboyedu.com
10.0.0.103 21 91 0 0.00 0.01 0.05 cdfhilmrstw - elk103.oldboyedu.com
rootelk101 ~]# jps
3572 Jps
3371 Elasticsearch
[rootelk101 ~]# ps -ef|grep java
oldboye 3371 1 99 14:02 ? 00:00:35 /oldboyedu/softwares/jdk1.8.0_291/bin/java -Xshare:auto -Des.networkaddress.cache.ttl60 -Des.networkaddress.cache.negative.ttl10 -XX:AlwaysPreTouch -Xss1m -Djava.awt.headlesstrue -Dfile.encodingUTF-8 -Djna.nosystrue -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafetrue -Dio.netty.noKeySetOptimizationtrue -Dio.netty.recycler.maxCapacityPerThread0 -Dio.netty.allocator.numDirectArenas0 -Dlog4j.shutdownHookEnabledfalse -Dlog4j2.disable.jmxtrue -Dlog4j2.formatMsgNoLookupstrue -Djava.locale.providersSPI,JRE -XX:UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction75 -XX:UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir/tmp/elasticsearch-1143701341521397671 -XX:HeapDumpOnOutOfMemoryError -XX:HeapDumpPathdata -XX:ErrorFilelogs/hs_err_pid%p.log -XX:PrintGCDetails -XX:PrintGCDateStamps -XX:PrintTenuringDistribution -XX:PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:UseGCLogFileRotation -XX:NumberOfGCLogFiles32 -XX:GCLogFileSize64m -Xms1024m -Xmx1024m -XX:MaxDirectMemorySize536870912 -Des.path.home/oldboyedu/softwares/es7/elasticsearch-7.17.5 -Des.path.conf/oldboyedu/softwares/es7/elasticsearch-7.17.5/config -Des.distribution.flavordefault -Des.distribution.typetar -Des.bundled_jdktrue -cp /oldboyedu/softwares/es7/elasticsearch-7.17.5/lib/* org.elasticsearch.bootstrap.Elasticsearch
root 3590 2034 0 14:03 pts/0 00:00:00 grep --colorauto java(7)elk101同步配置到其他节点
data_rsync.sh /etc/profile.d/jdk.sh
data_rsync.sh /usr/lib/systemd/system/es7.service
data_rsync.sh /oldboyedu/softwares/jdk1.8.0_291(8)其他节点重启服务
systemctl daemon-reload systemctl restart es73.5 修改ES环境的堆(heap)内存大小
(1)elk101修改配置文件
[rootelk101.oldboyedu.com ~]# vim /oldboyedu/softwares/es7/elasticsearch-7.17.5/config/jvm.options
...
-Xms256m
-Xmx256m(2)elk101重启服务并验证堆内存大小若不配置默认1GB
[rootelk101 ~]# systemctl restart es7
[rootelk101 ~]# jps
3987 Jps
3753 Elasticsearch
[rootelk101 ~]# jmap -heap 3753|grep MaxHeapSizeMaxHeapSize 268435456 (256.0MB)(3)elk101同步环境到其他节点
data_rsync.sh /oldboyedu/softwares/es7/elasticsearch-7.17.5/config/jvm.options(4)其他节点重启服务并验证堆内存大小
[rootelk102.oldboyedu.com ~]# systemctl restart es7[rootelk102.oldboyedu.com ~]# jmap -heap jps | awk /Elasticsearch/{print $1} | grep MaxHeapSizeMaxHeapSize 268435456 (256.0MB)3.6 ES集群的多实例部署
需要解决的问题 程序安装目录数据目录日志目录端口冲突9200|9300
(1)所有节点准备ES6的工作目录
install -d /oldboyedu/{data,logs,softwares}/es6 -o oldboyedu -g oldboyedu(2)elk101下载ES6软件包
curl -o elasticsearch-6.8.23.tar.gz http://192.168.15.253/ElasticStack/softwares/binary/elasticsearch-6.8.23.tar.gz(3)elk101解压软件包 tar xf elasticsearch-6.8.23.tar.gz -C /oldboyedu/softwares/es6/ chown oldboyedu:oldboyedu -R /oldboyedu/softwares/es6/elasticsearch-6.8.23/ (4)elk101修改配置文件
[rootelk101.oldboyedu.com ~]# yy /oldboyedu/softwares/es6/elasticsearch-6.8.23/config/elasticsearch.yml
cluster.name: oldboyedu-linux85-es6
node.name: elk101.oldboyedu.com
path.data: /oldboyedu/data/es6
path.logs: /oldboyedu/logs/es6
network.host: 0.0.0.0
http.port: 19200
transport.tcp.port: 19300
discovery.zen.ping.unicast.hosts: [elk101.oldboyedu.com,elk102.oldboyedu.com,elk103.oldboyedu.com]
discovery.zen.minimum_master_nodes: 2
[rootelk101.oldboyedu.com ~]# (5)elk101修改堆内存大小
[rootelk101.oldboyedu.com ~]# vim /oldboyedu/softwares/es6/elasticsearch-6.8.23/config/jvm.options
...
-Xms256m
-Xmx256m(6)elk101编写启动脚本
cat /usr/lib/systemd/system/es6.service EOF
[Unit]
Descriptionoldboyedu linux85 es6
Afternetwork.target[Service]
Typesimple
EnvironmentJAVA_HOME/oldboyedu/softwares/jdk1.8.0_291
ExecStart/oldboyedu/softwares/es6/elasticsearch-6.8.23/bin/elasticsearch
Useroldboyedu
LimitNOFILE131070[Install]
WantedBymulti-user.target
EOF(7)elk101同步配置文件
data_rsync.sh /oldboyedu/softwares/es6/
data_rsync.sh /usr/lib/systemd/system/es6.service(8)修改各节点的配置文件
[rootelk102.oldboyedu.com ~]# vim /oldboyedu/softwares/es6/elasticsearch-6.8.23/config/elasticsearch.yml
...
node.name: elk102.oldboyedu.com[rootelk103.oldboyedu.com ~]# vim /oldboyedu/softwares/es6/elasticsearch-6.8.23/config/elasticsearch.yml
...
node.name: elk103.oldboyedu.com(9)启动服务
systemctl daemon-reload systemctl enable --now es6(10)查看服务状态
[rootelk101.oldboyedu.com ~]# curl 10.0.0.101:19200/_cat/nodes
10.0.0.102 50 77 1 0.58 0.25 0.14 mdi - elk102.oldboyedu.com
10.0.0.101 59 82 1 0.37 0.25 0.13 mdi * elk101.oldboyedu.com
10.0.0.103 44 76 42 0.46 0.22 0.12 mdi - elk103.oldboyedu.com查看集群节点的API
curl 10.0.0.101:9200/_cat/nodes
[rootelk101 ~]# curl 10.0.0.101:9200/_cat/nodes
10.0.0.103 43 75 0 0.00 0.01 0.05 cdfhilmrstw - elk103.oldboyedu.com
10.0.0.101 35 80 0 0.02 0.03 0.05 cdfhilmrstw - elk101.oldboyedu.com
10.0.0.102 57 84 0 0.00 0.01 0.05 cdfhilmrstw * elk102.oldboyedu.com
查看集群状态:
curl 10.0.0.101:9200/_cluster/health 2/dev/null | jq
[rootelk101 ~]# curl 10.0.0.101:9200/_cluster/health 2/dev/null | jq
{cluster_name: oldboyedu-linux85-binary,status: green,timed_out: false,number_of_nodes: 3,number_of_data_nodes: 3,active_primary_shards: 2,active_shards: 4,relocating_shards: 0,initializing_shards: 0,unassigned_shards: 0,delayed_unassigned_shards: 0,number_of_pending_tasks: 0,number_of_in_flight_fetch: 0,task_max_waiting_in_queue_millis: 0,active_shards_percent_as_number: 100
}3.7 常见报错
1.java.lang.RuntimeException: can not run elasticsearch as root
报错原因:不能以root用户启动ES服务。
解决方案:使用普通用户启动服务即可。2.bootstrap check failure [1] of [3]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
报错原因:程序默认的的文件打开数量上线过低。
解决方案:调大文件打开数量上限即可。3.bootstrap check failure [2] of [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
报错原因内核参数虚拟内存映射过低。
解决方案调大内核虚拟内存映射值即可。4.bootstrap check failure [3] of [3]: max number of threads [3795] for user [oldboyedu] is too low, increase to at least [4096]
报错原因程序打开的线程数量设置过低。
解决方案:调大程序打开的线程数量即可。5.initial heap size [268435456] not equal to maximum heap size [1031798784]; this can cause resize pauses and prevents mlockall from locking the entire heap
报错原因初始化堆内存和最大堆内存大小不一致。
解决方案:观察配置是否生效建议将-Xms和-Xmx值配置一致。今日作业:
完成课堂的所有练习并整理思维导图;使用zabbix监控ES服务
扩展作业: - 使用ansible的playbook一键搭建ES6和ES7多实例服务
