怎么免费建立自己的网站平台,宁波江北区城市建设档案馆网站,关键词点击排名系统,网页设计模板素材简单文章目录 安装jdk配置主机名Zookeeper 下载与安装Kafka 下载与安装测试集群版安装测试输出 安装jdk
略
配置主机名 hostnamectl set-hostname kafka_1 /etc/sysconfig/network
HOSTNAMEkafka_1/etc/hosts
ip kafka_1ping kafka_1 测试
Zookeeper 下载与安装
由于 集群… 文章目录 安装jdk配置主机名Zookeeper 下载与安装Kafka 下载与安装测试集群版安装测试输出 安装jdk
略
配置主机名 hostnamectl set-hostname kafka_1 /etc/sysconfig/network
HOSTNAMEkafka_1/etc/hosts
ip kafka_1ping kafka_1 测试
Zookeeper 下载与安装
由于 集群中的 Leader 的监控 和 Topic 元数据 存储在 Zookeeper 中 所以需要安装 zkhttps://zookeeper.apache.org/releases.html 中选择需要的版本conf 目录 复制zoo_sample.cfg 为 zoo.cfg, 修改其中的配置 dataDir 指定自己的目录启动zk使用 zkServer.sh start zoo.cfg
cd /opt/pkg/
wget --no-check-certificate https://dlcdn.apache.org/zookeeper/zookeeper-3.8.2/apache-zookeeper-3.8.2-bin.tar.gz
tar zxf apache-zookeeper-3.8.2-bin.tar.gz
mv apache-zookeeper-3.8.2-bin ../app/
cd /opt/app/apache-zookeeper-3.8.2-bin/
cp zoo_sample.cfg zoo.cfg
./bin/zkServer.sh start zoo.cfg---
[rootloaclhost bin]# ./zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/app/apache-zookeeper-3.8.2-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: standalone
Kafka 下载与安装
https://kafka.apache.org/downloads
cd /opt/pkg/
wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz
tar zxf kafka_2.13-3.5.1.tgz
mv kafka_2.13-3.5.1 ../app/
cd /opt/app
cd kafka_2.13-3.5.1/
cd config/
cp server.properties server.properties.bak
vim server.properties
---
listenersPLAINTEXT://kafka_1:9092
og.dirs./kafka-logs
zookeeper.connectkafka_1:2181
---cd ../
./bin/kafka-server-start.sh -daemon config/server.properties配置 config/server.properties
broker.id
listenersPLAINTEXT://主机名:9092
log.dirs # 日志目录
zookeeper.connect主机名:2181启动./bin/kafka-server-start.sh -daemon config/server.properties停止./bin/kafka-server-stop.sh
测试
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092 --create --topic topic_01 --partitions 3 --replication-factor 1 # 单机版副本因子为1 多了没有意义
./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092 --topic topic_01 --group g1 # 消费者 查看输出
./bin/kafka-console-producer.sh --broker-list kafka_1:9092 --topic topic_01 # 开始输入rootloaclhost kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092 --create --topic topic_0 OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should c WARNING: Due to limitations in metric names, topics with a period (‘.’) or underscore () could collide. To avo Created topic topic_01. [rootloaclhost kafka_2.13-3.5.1]# ./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092 --topic topic OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should c rootloaclhost kafka_2.13-3.5.1]# ./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092 --topic topic_01 --group g1 OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN 集群版安装
使用3台主机进行搭建三台机器上都需要安装 jdkZookeeperKafka其中 /etc/hosts 都需要配置好三台主机的ip 主机名映射需要额外注意的是三台主机需要做一次时间同步
yum -y install ntpdate
timedatectl set-timezone Asia/Shanghai
ntpdate ntp1.aliyun.com # aliyun 有ntp1~7 或者 cn.pool.ntp.org复制文件到 另外两台主机
[rootkafka_1 pkg]# pwd
/opt/pkg
scp kafka_2.13-3.5.1.tgz rootkafka_2:/opt/pkg
scp kafka_2.13-3.5.1.tgz rootkafka_3:/opt/pkg
scp apache-zookeeper-3.8.2-bin.tar.gz rootkafka_3:/opt/pkg
scp apache-zookeeper-3.8.2-bin.tar.gz rootkafka_2:/opt/pkg
配置 zoo.cfg:
# 数据目录 每个机器中此目录下 需要创建 myid文件内容 和server.id 对应 , 可以指定一个绝对目录
dataDir/opt/app/cluster/apache-zookeeper-3.8.2/dataDir server.1kafka_1:2888:3888
server.2kafka_2:2888:3888
server.3kafka_3:2888:3888
4lw.commands.whitelist*myid生成 echo 1 /opt/app/cluster/apache-zookeeper-3.8.2/dataDir/myid # 1,2,3 配置 config/server.properties
broker.id0 # 需要修改 1,2
listenersPLAINTEXT://kafka_1:9092 # 主机名和当前机器对应
log.dirs/opt/app/cluster/kafka_2.13-3.5.1/log_dirs # 日志目录
zookeeper.connectkafka_1:2181,kafka_2:2181,kafka_3:2181启动zk 和 kafka:
cd /opt/app/cluster/apache-zookeeper-3.8.2/
./bin/zkServer.sh start zoo.cfg
cd /opt/app/cluster/kafka_2.13-3.5.1
./bin/kafka-server-start.sh -daemon config/server.properties报错 Using config: /opt/app/cluster/apache-zookeeper-3.8.2/bin/…/conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Error contacting service. It is probably not running. 原因是防火墙没关 systemctl status firewalld
#● firewalld.service - firewalld - dynamic firewall daemon
# Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
# Active: active (running) since Sun 2023-08-27 19:48:04 CST; 2h 32min ago
# Docs: man:firewalld(1)
# Main PID: 649 (firewalld)
# CGroup: /system.slice/firewalld.service
# └─649 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
#
systemctl disable firewalld测试
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --create --topic topic01 --partitions 3 --replication-factor 3 # 创建 topic
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --create --topic topic02 --partitions 3 --replication-factor 3 # 创建 topic
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list # 查看
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --describe --topic topic02 # 查看详细描述
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --alter --topic topic02 --partitions 4 # 修改分区数量,只能增不能减设置小于3则报错
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --delete --topic topic02 # 删除topic---
# 消息的生产与消费/订阅
./bin/kafka-console-producer.sh --broker-list kafka_1:9092,kafka_2:9092,kafka_3:9092 --topic topic01 # 生产
./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --topic topic01 --group g1 --property print.keytrue --property print.valuetrue --property key.separator, # 消费
---
# 查看消费组信息
./bin/kafka-consumer-groups.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list g1 # 查看
./bin/kafka-consumer-groups.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --describe --group g1 # 查看详情输出
[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --create --topic topic01 --partitions 3 --replication-factor 3
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN
Created topic topic01.
[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --create --topic topic02 --partitions 3 --replication-factor 3
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN
Created topic topic02.
[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN
topic01
topic02[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --describe --topic topic02
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN
Topic: topic02 TopicId: _s44mBQqTLaSmVWcZA2GTw PartitionCount: 3 ReplicationFactor: 3 Configs:Topic: topic02 Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,1Topic: topic02 Partition: 1 Leader: 1 Replicas: 1,2,0 Isr: 1,2Topic: topic02 Partition: 2 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --alter --topic topic02 --partitions 4[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --describe --topic topic02
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN
Topic: topic02 TopicId: _s44mBQqTLaSmVWcZA2GTw PartitionCount: 4 ReplicationFactor: 3 Configs:Topic: topic02 Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,1Topic: topic02 Partition: 1 Leader: 1 Replicas: 1,2,0 Isr: 1,2Topic: topic02 Partition: 2 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2Topic: topic02 Partition: 3 Leader: 2 Replicas: 2,1,0 Isr: 2,1[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN
topic01
topic02
[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --delete --topic topic02
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN
[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN
topic01---
# 生产者
[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-console-producer.sh --broker-list kafka_1:9092,kafka_2:9092,kafka_3:9092 --topic topic01
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsN
123
123
key123
# 消费者
[rootkafka_1 kafka_2.13-3.5.1]# ./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --topic topic01 --group g1 --property print.keytrue --property print.valuetrue --property key.separator,
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreadsNnull,123
null,123
null,key123
