简洁的企业网站源码,vi品牌设计,雅诗兰黛网络营销策划方案,深圳商城网站哪家做的好Win10环境借助DockerDesktop部署最新版大数据时序数据库Apache Druid32.0.0
前言
大数据分析中#xff0c;有一种常见的场景#xff0c;那就是时序数据#xff0c;简言之#xff0c;数据一旦产生绝对不会修改#xff0c;随着时间流逝#xff0c;每个时间点都会有个新的…Win10环境借助DockerDesktop部署最新版大数据时序数据库Apache Druid32.0.0
前言
大数据分析中有一种常见的场景那就是时序数据简言之数据一旦产生绝对不会修改随着时间流逝每个时间点都会有个新的状态值。这种时序数据的量级往往异常夸张例如传感器的原始监控数据
https://lizhiyong.blog.csdn.net/article/details/114898620
一个简单的加速度传感器一年的数据量就是31e制造业传感器数据如果不经底层PLC等下位机预处理直接打到边缘计算网关即使mqtt也会有巨大的负载
类似的还有服务器的原始监控数据例如常见的Prometheus和Zabbix当集群很多时监控项同样很多再算上虚拟化后的容器和虚拟机内都可能部署了监控此时的数据量级就灰常可观一小时几百亿条数据都是常见的事情
但是很多原始的监控数据如果全部存下来存储成本高的可怕同时信息密度极低更多时候我们可能只关注近期的全部热数据来做在线的模型训练人工查看每秒钟几千条数据也是不切合实际的事实上做一个简单的秒级/分钟级统计就能满足大多数的分析场景超过1天的冷数据其实已经没什么时效性。
对于此类场景可以高吞吐、预聚合的数据库在压测后从Apache Druid、Clickhouse、Kylin中选择了前者。。。专业的事情要交给专业的组件去做
对于非内核和二开的业务开发人员更多场景应该关注的是API、特性及用法不应该在部署这种事情上花费太多精力笔者之前已部署了Docker Desktop
https://lizhiyong.blog.csdn.net/article/details/145580868
今天在Win10环境再搭建个Apache Druid最新版玩玩。
版本选择
官网
https://druid.apache.org/注意不是阿里数据库连接池的那个Druid 截至2025-02-13Apache Druid最新版本是32.0.0。
资源准备
参考官网
https://druid.apache.org/docs/latest/tutorials/docker官方给出了使用docker-compose.yml编排容器的教程作为一个实时组件大内存是必须的但是启动8个容器【ZookeeperPostgreSQL6个Druid】每个最多7GB内存也不是什么大事
https://raw.githubusercontent.com/apache/druid/32.0.0/distribution/docker/docker-compose.yml获取到这个资源文件
version: 2.2volumes:metadata_data: {}middle_var: {}historical_var: {}broker_var: {}coordinator_var: {}router_var: {}druid_shared: {}services:postgres:container_name: postgresimage: postgres:latestports:- 5432:5432volumes:- metadata_data:/var/lib/postgresql/dataenvironment:- POSTGRES_PASSWORDFoolishPassword- POSTGRES_USERdruid- POSTGRES_DBdruid# Need 3.5 or later for container nodeszookeeper:container_name: zookeeperimage: zookeeper:3.5.10ports:- 2181:2181environment:- ZOO_MY_ID1coordinator:image: apache/druid:32.0.0container_name: coordinatorvolumes:- druid_shared:/opt/shared- coordinator_var:/opt/druid/vardepends_on:- zookeeper- postgresports:- 8081:8081command:- coordinatorenv_file:- environmentbroker:image: apache/druid:32.0.0container_name: brokervolumes:- broker_var:/opt/druid/vardepends_on:- zookeeper- postgres- coordinatorports:- 8082:8082command:- brokerenv_file:- environmenthistorical:image: apache/druid:32.0.0container_name: historicalvolumes:- druid_shared:/opt/shared- historical_var:/opt/druid/vardepends_on: - zookeeper- postgres- coordinatorports:- 8083:8083command:- historicalenv_file:- environmentmiddlemanager:image: apache/druid:32.0.0container_name: middlemanagervolumes:- druid_shared:/opt/shared- middle_var:/opt/druid/vardepends_on: - zookeeper- postgres- coordinatorports:- 8091:8091- 8100-8105:8100-8105command:- middleManagerenv_file:- environmentrouter:image: apache/druid:32.0.0container_name: routervolumes:- router_var:/opt/druid/vardepends_on:- zookeeper- postgres- coordinatorports:- 3012:8888 #这里笔者改为3012防止霸占有用的端口command:- routerenv_file:- environment参照官网另一篇
https://druid.apache.org/docs/latest/configuration/自己玩玩可以先不改这些运行时配置容器启动的后续要重新部署也非常容易
还需要
https://raw.githubusercontent.com/apache/druid/32.0.0/distribution/docker/environment做另一个配置文件
# Java tuning
#DRUID_XMX1g
#DRUID_XMS1g
#DRUID_MAXNEWSIZE250m
#DRUID_NEWSIZE250m
#DRUID_MAXDIRECTMEMORYSIZE6172m
DRUID_SINGLE_NODE_CONFmicro-quickstartdruid_emitter_logging_logLeveldebugdruid_extensions_loadList[druid-histogram, druid-datasketches, druid-lookups-cached-global, postgresql-metadata-storage, druid-multi-stage-query]druid_zk_service_hostzookeeperdruid_metadata_storage_host
druid_metadata_storage_typepostgresql
druid_metadata_storage_connector_connectURIjdbc:postgresql://postgres:5432/druid
druid_metadata_storage_connector_userdruid
druid_metadata_storage_connector_passwordFoolishPassworddruid_indexer_runner_javaOptsArray[-server, -Xmx1g, -Xms1g, -XX:MaxDirectMemorySize3g, -Duser.timezoneUTC, -Dfile.encodingUTF-8, -Djava.util.logging.managerorg.apache.logging.log4j.jul.LogManager]
druid_indexer_fork_property_druid_processing_buffer_sizeBytes256MiBdruid_storage_typelocal
druid_storage_storageDirectory/opt/shared/segments
druid_indexer_logs_typefile
druid_indexer_logs_directory/opt/shared/indexing-logsdruid_processing_numThreads2
druid_processing_numMergeBuffers2DRUID_LOG4J?xml version1.0 encodingUTF-8 ?Configuration statusWARNAppendersConsole nameConsole targetSYSTEM_OUTPatternLayout pattern%d{ISO8601} %p [%t] %c - %m%n//Console/AppendersLoggersRoot levelinfoAppenderRef refConsole//RootLogger nameorg.apache.druid.jetty.RequestLog additivityfalse levelDEBUGAppenderRef refConsole//Logger/Loggers/Configuration部署文件看起来麻雀虽小五脏俱全
部署
PS C:\Users\zhiyong cd E:\dockerData\volume\druid1
PS E:\dockerData\volume\druid1 ls目录: E:\dockerData\volume\druid1Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 2025-02-13 23:26 2980 docker-compose.yml
-a---- 2025-02-13 23:33 1576 environment
PS E:\dockerData\volume\druid1 docker compose up -d
time2025-02-13T23:34:3908:00 levelwarning msgE:\\dockerData\\volume\\druid1\\docker-compose.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion
[] Running 72/15✔ router Pulled 230.7s ✔ coordinator Pulled 230.7s ✔ postgres Pulled 181.0s ✔ historical Pulled 230.7s ✔ broker Pulled 230.7s ✔ middlemanager Pulled 230.7s ✔ zookeeper Pulled 85.7s [] Running 15/15✔ Network druid1_default Created 0.1s ✔ Volume druid1_druid_shared Created 0.0s ✔ Volume druid1_historical_var Created 0.0s ✔ Volume druid1_middle_var Created 0.0s ✔ Volume druid1_router_var Created 0.0s ✔ Volume druid1_metadata_data Created 0.0s ✔ Volume druid1_coordinator_var Created 0.0s ✔ Volume druid1_broker_var Created 0.0s ✔ Container postgres Started 2.4s ✔ Container zookeeper Started 2.4s ✔ Container coordinator Started 1.6s ✔ Container router Started 2.5s ✔ Container broker Started 2.3s ✔ Container historical Started 2.5s ✔ Container middlemanager Started 2.8s
PS E:\dockerData\volume\druid1拉取镜像成功后很快就能拉起容器 好家伙。。。还顺便把其它组件的端口也给暴露出来了。。。 于是还**白piao**到一个PG和Zookeeper
验证
http://localhost:3012/unified-console.html#灰常好现在已经拥有了一个最新Apache Druid32.0.0
转载请注明出处https://lizhiyong.blog.csdn.net/article/details/145622903
