成都房地产信息网官网,上海seo有哪些公司,颍上网站建设,石家庄网站建设诚荐优度网络进入正文前#xff0c;感谢宝子们订阅专题、点赞、评论、收藏#xff01;关注IT贫道#xff0c;获取高质量博客内容#xff01; #x1f3e1;个人主页#xff1a;含各种IT体系技术#xff0c;IT贫道_Apache Doris,大数据OLAP体系技术栈,Kerberos安全认证-CSDN博客 … 进入正文前感谢宝子们订阅专题、点赞、评论、收藏关注IT贫道获取高质量博客内容 个人主页含各种IT体系技术IT贫道_Apache Doris,大数据OLAP体系技术栈,Kerberos安全认证-CSDN博客 订阅拥抱独家专题你的订阅将点燃我的创作热情 点赞赞同优秀创作你的点赞是对我创作最大的认可 ⭐️ 收藏收藏原创博文让我们一起打造IT界的荣耀与辉煌 ✏️评论留下心声墨迹你的评论将是我努力改进的方向 目录
1. Java 读写ClickHouse API
2. Spark 写入 ClickHouse API 1. Java 读写ClickHouse API
Java读取ClickHouse中的数据API 。
1 首先需要加入maven依赖
!-- 连接ClickHouse需要驱动包--
dependencygroupIdru.yandex.clickhouse/groupIdartifactIdclickhouse-jdbc/artifactIdversion0.2.4/version
/dependency
2) Java 读取ClickHouse单节点表数据
ClickHouseProperties props new ClickHouseProperties();
props.setUser(default);
props.setPassword();
BalancedClickhouseDataSource dataSource new BalancedClickhouseDataSource(jdbc:clickhouse://node1:8123/default, props);
ClickHouseConnection conn dataSource.getConnection();
ClickHouseStatement statement conn.createStatement();
ResultSet rs statement.executeQuery(select id,name,age from test);
while(rs.next()){int id rs.getInt(id);String name rs.getString(name);int age rs.getInt(age);System.out.println(id id,name name ,age age);
}
3. Java 读取ClickHouse集群表数据
ClickHouseProperties props new ClickHouseProperties();
props.setUser(default);
props.setPassword();
BalancedClickhouseDataSource dataSource new BalancedClickhouseDataSource(jdbc:clickhouse://node1:8123,node2:8123,node3:8123/default, props);
ClickHouseConnection conn dataSource.getConnection();
ClickHouseStatement statement conn.createStatement();
ResultSet rs statement.executeQuery(select id,name from t_cluster);
while(rs.next()){int id rs.getInt(id);String name rs.getString(name);System.out.println(id id,name name );
}
4) Java向ClickHouse 表中写入数据。
# API 操作ClickHouseProperties props new ClickHouseProperties();
props.setUser(default);
props.setPassword();
BalancedClickhouseDataSource dataSource new BalancedClickhouseDataSource(jdbc:clickhouse://node1:8123/default, props);
ClickHouseConnection conn dataSource.getConnection();
ClickHouseStatement statement conn.createStatement();
statement.execute(insert into test values (100,王五,30));//可以拼接批量插入多条#查询default库下 test表 数据node1 :) select * from test;┌──id─┬─name─┬─age─┐│ 100 │ 王五 │ 30 │└─────┴──────┴─────┘┌─id─┬─name─┬─age─┐│ 1 │ 张三 │ 18 ││ 2 │ 李四 │ 19 │└────┴──────┴─────┘ 2. Spark 写入 ClickHouse API
SparkCore写入ClickHouse可以直接采用写入方式。下面案例是使用SparkSQL将结果存入ClickHouse对应的表中。在ClickHouse中需要预先创建好对应的结果表。
1) 导入依赖
!-- 连接ClickHouse需要驱动包--
dependencygroupIdru.yandex.clickhouse/groupIdartifactIdclickhouse-jdbc/artifactIdversion0.2.4/version!-- 去除与Spark 冲突的包 --exclusionsexclusiongroupIdcom.fasterxml.jackson.core/groupIdartifactIdjackson-databind/artifactId/exclusionexclusiongroupIdnet.jpountz.lz4/groupIdartifactIdlz4/artifactId/exclusion
/exclusions
/dependency!-- Spark-core --
dependencygroupIdorg.apache.spark/groupIdartifactIdspark-core_2.11/artifactIdversion2.3.1/version
/dependency
!-- SparkSQL --
dependencygroupIdorg.apache.spark/groupIdartifactIdspark-sql_2.11/artifactIdversion2.3.1/version
/dependency
!-- SparkSQL ON Hive--
dependencygroupIdorg.apache.spark/groupIdartifactIdspark-hive_2.11/artifactIdversion2.3.1/version
/dependency
2) 代码编写
val session: SparkSession SparkSession.builder().master(local).appName(test).getOrCreate()
val jsonList List[String]({\id\:1,\name\:\张三\,\age\:18},{\id\:2,\name\:\李四\,\age\:19},{\id\:3,\name\:\王五\,\age\:20}
)//将jsonList数据转换成DataSet
import session.implicits._
val ds: Dataset[String] jsonList.toDS()val df: DataFrame session.read.json(ds)
df.show()//将结果写往ClickHouse
val url jdbc:clickhouse://node1:8123/default
val table test
val properties new Properties()
properties.put(driver, ru.yandex.clickhouse.ClickHouseDriver)
properties.put(user, default)
properties.put(password, )
properties.put(socket_timeout, 300000)
df.write.mode(SaveMode.Append).option(JDBCOptions.JDBC_BATCH_INSERT_SIZE, 100000).jdbc(url, table, properties) 如需博文中的资料请私信博主。
