东莞市研发网站建设品牌,网站开发有哪些流程,vs加数据库做网站,工程造价信息网南京文章目录 主机准备部署编辑 docker-compose.ymlcontrollerbroker生成cluster_id 一篇完整的 docker-compose.yml 文件查看集群状态使用 kafka-ui 查看拉取 kafka-ui添加集群查看集群状态 使用命令行查看 配置讲解controllerbroker 主机准备
IPcontroller idbroker id192.168.1… 文章目录 主机准备部署编辑 docker-compose.ymlcontrollerbroker生成cluster_id 一篇完整的 docker-compose.yml 文件查看集群状态使用 kafka-ui 查看拉取 kafka-ui添加集群查看集群状态 使用命令行查看 配置讲解controllerbroker 主机准备
IPcontroller idbroker id192.168.142.15715192.168.142.15626192.168.142.15537192.168.142.15848
部署
创建文件夹
mkdir -p /docker/kafka/{data,data1,data2}chmod 777 -R {data,data1,data2}编辑 docker-compose.yml
controller
docker-compose.yml
services:controller:image: bitnami/kafka:3.8.0container_name: controllernetwork_mode: hostrestart: alwaysvolumes:- /docker/kafka/data:/data- /etc/localtime:/etc/localtime:roenvironment:KAFKA_CFG_NODE_ID: 1KAFKA_CFG_PROCESS_ROLES: controllerKAFKA_CFG_LISTENERS: CONTROLLER://192.168.142.157:9093KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXTKAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLERKAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1192.168.142.157:9093,2192.168.142.156:9093,3192.168.142.155:9093,4192.168.142.158:9093KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0KAFKA_CFG_LOG_DIRS: /databroker
image: bitnami/kafka:3.8.0container_name: brokernetwork_mode: hostrestart: alwaysvolumes:- /etc/localtime:/etc/localtime:ro- /docker/kafka/data1:/data1- /docker/kafka/data2:/data2environment:KAFKA_CFG_NODE_ID: 5KAFKA_CFG_PROCESS_ROLES: brokerKAFKA_CFG_LISTENERS: PLAINTEXT://192.168.142.157:9092KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.142.157:9092KAFKA_CFG_INTER_BROKER_LISTENER_NAME: PLAINTEXTKAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLERKAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXTKAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1192.168.142.157:9093,2192.168.142.156:9093,3192.168.142.155:9093,4192.168.142.158:9093KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0KAFKA_CFG_LOG_DIRS: /data1,/data2所有的主机这一段基本都是一样的 只有 id 不同id 可以根据上面表格进行修改 生成cluster_id
rootmaster:/docker/kafka# docker run -it --rm bitnami/kafka:3.8.0 kafka-storage.sh random-uuid
kafka 06:43:36.79 INFO
kafka 06:43:36.80 INFO Welcome to the Bitnami kafka container
kafka 06:43:36.80 INFO Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 06:43:36.80 INFO Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 06:43:36.80 INFO Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
kafka 06:43:36.80 INFO PavLImfXSqaOOdrMgq1teA拿到 id PavLImfXSqaOOdrMgq1teA 新增到 docker-compose.yml 文件中
一篇完整的 docker-compose.yml 文件
services:controller:image: bitnami/kafka:3.8.0container_name: controllernetwork_mode: hostrestart: alwaysvolumes:- /docker/kafka/data:/data- /etc/localtime:/etc/localtime:roenvironment:KAFKA_CFG_NODE_ID: 1KAFKA_CFG_PROCESS_ROLES: controllerKAFKA_CFG_LISTENERS: CONTROLLER://192.168.142.157:9093KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXTKAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLERKAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1192.168.142.157:9093,2192.168.142.156:9093,3192.168.142.155:9093,4192.168.142.158:9093KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0KAFKA_CFG_LOG_DIRS: /dataKAFKA_KRAFT_CLUSTER_ID: PavLImfXSqaOOdrMgq1teAbroker:image: bitnami/kafka:3.8.0container_name: brokernetwork_mode: hostrestart: alwaysvolumes:- /etc/localtime:/etc/localtime:ro- /docker/kafka/data1:/data1- /docker/kafka/data2:/data2environment:KAFKA_CFG_NODE_ID: 5KAFKA_CFG_PROCESS_ROLES: brokerKAFKA_CFG_LISTENERS: PLAINTEXT://192.168.142.157:9092KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.142.157:9092KAFKA_CFG_INTER_BROKER_LISTENER_NAME: PLAINTEXTKAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLERKAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXTKAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1192.168.142.157:9093,2192.168.142.156:9093,3192.168.142.155:9093,4192.168.142.158:9093KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0KAFKA_CFG_LOG_DIRS: /data1,/data2KAFKA_KRAFT_CLUSTER_ID: PavLImfXSqaOOdrMgq1teA启动
docker compose up -d查看启动状态
docker ps -a | grep Up | wc -l
2查看集群状态
使用 kafka-ui 查看
拉取 kafka-ui
只需要在一台主机上部署 ui 就可以了
docker run -d -p 8080:8080 -e DYNAMIC_CONFIG_ENABLEDtrue provectuslabs/kafka-ui:v0.7.1 通过 ip 8080 端口访问 添加集群 查看集群状态 使用命令行查看
rootmaster:/docker/kafka# docker exec -it broker ./opt/bitnami/kafka/bin/kafka-metadata-quorum.sh --bootstrap-controller 192.168.142.157:9093,192.168.142.156:9093,192.168.142.155:9093,192.168.142.158:9093 describe --status
ClusterId: PavLImfXSqaOOdrMgq1teA
LeaderId: 1
LeaderEpoch: 22
HighWatermark: 7695
MaxFollowerLag: 0
MaxFollowerLagTimeMs: 0
CurrentVoters: [1,2,3,4]
CurrentObservers: [5,6,7,8]到此集群搭建完成
配置讲解
controller
参数解释KAFKA_CFG_NODE_ID节点 idKAFKA_CFG_PROCESS_ROLES节点角色KAFKA_CFG_LISTENERS节点监听的 ip 和端口并且标记为 controllerKAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP定义不同监听器使用的加密协议KAFKA_CFG_CONTROLLER_LISTENER_NAMES控制器Controller用于接收来自其他控制器节点和 broker 节点请求的监听器Listener的名称KAFKA_CFG_CONTROLLER_QUORUM_VOTERS添加集群节点KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS当一个新的消费者加入到一个空的消费组时协调者Coordinator将延迟多长时间再开始重新平衡Rebalance的过程KAFKA_CFG_LOG_DIRS日志目录KAFKA_KRAFT_CLUSTER_ID集群 id
broker
参数解释KAFKA_CFG_NODE_ID节点 idKAFKA_CFG_PROCESS_ROLES节点角色KAFKA_CFG_LISTENERS节点监听的 ip 和端口并且标记为 brokerKAFKA_CFG_INTER_BROKER_LISTENER_NAME集群中 broker 之间通信所使用的监听器名称KAFKA_CFG_CONTROLLER_LISTENER_NAMES监听器的名称KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP定义不同监听器使用的加密协议KAFKA_CFG_CONTROLLER_QUORUM_VOTERS添加集群节点KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS当一个新的消费者加入到一个空的消费组时协调者Coordinator将延迟多长时间再开始重新平衡Rebalance的过程KAFKA_CFG_LOG_DIRS日志目录KAFKA_KRAFT_CLUSTER_ID集群 id
