当前位置: 首页 > news >正文

多种郑州网站建设编程培训机构加盟哪家好

多种郑州网站建设,编程培训机构加盟哪家好,建e室内设计网贴图,wordpress标签图片缘起 今年(2023年) 2月的时候做了个适配Amazon S3对象存储接口的需求,由于4月份自学考试临近,一直在备考就拖着没总结记录下,开发联调过程中也出现过一些奇葩的问题,最近人刚从考试缓过来顺手记录一下。 S3对象存储的基本概念 …

缘起

今年(2023年) 2月的时候做了个适配Amazon S3对象存储接口的需求,由于4月份自学考试临近,一直在备考就拖着没总结记录下,开发联调过程中也出现过一些奇葩的问题,最近人刚从考试缓过来顺手记录一下。

S3对象存储的基本概念

S3是什么?

Amazon S3(Simple Storage Service)对象存储出现得比较早且使用简单的RESTful API,于是成为了对象存储服务(Object Storage Service,OSS)业内的标准接口规范。

S3的逻辑模型

如下图,我们可以把S3的存储空间想象成无限的,想存储一个任意格式的文件到S3服务中,只需要知道要把它放到哪个桶(Bucket)中,它的名字(Object Id)应该是什么。

按图中的模型,可简单理解为S3是由若干个桶(Bucket)组成,每个桶中包含若干个不同标识的对象(Object),还有就是统一的访问入口(RESTful API),这样基本就足够了。

Minio客户端方式操作S3

详细API文档:https://min.io/docs/minio/linux/developers/java/API.html

以下代码异常处理做了简化,真实使用时请注意捕获异常做处理。

引入依赖

Maven:

<dependency><groupId>io.minio</groupId><artifactId>minio</artifactId><version>8.5.2</version>
</dependency>

Gradle:

dependencies {implementation("io.minio:minio:8.5.2")
}

初始化客户端

private static final String HTTP_PROTOCOL = "http";private MinioClient minioClient;
private String endpoint = "http://192.168.0.8:9200";
private String accessKey = "testKey";
private String secretKey = "testSecretKey";public void init() throws MalformedURLException {URL endpointUrl = new URL(endpoint);try {// url上无端口号时,识别http为80端口,https为443端口int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;//@formatter:offthis.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security).credentials(accessKey, secretKey).build();//@formatter:on// 忽略证书校验,防止自签名证书校验失败导致无法建立连接this.minioClient.ignoreCertCheck();} catch (Exception e) {e.printStackTrace();}
}

建桶

public boolean createBucket(String bucket) {try {minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());} catch (Exception e) {e.printStackTrace();return false;}return true;
}

删桶

public boolean deleteBucket(String bucket) {try {minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());logger.info("删除桶[{}]成功", bucket);} catch (Exception e) {e.printStackTrace();return false;}return true;
}

判断桶是否存在

public boolean bucketExists(String bucket) {try {return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());} catch (Exception e) {e.printStackTrace();return false;}
}

上传对象

public void upload(String bucket, String objectId, InputStream input) {try {//@formatter:offminioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId).stream(input, input.available(), -1).build());//@formatter:on} catch (Exception e) {e.printStackTrace();}
}

下载对象

提供两个下载方法,一个将输入流返回,另一个用参数输出流写出

public InputStream download(String bucket, String objectId) {try {return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {e.printStackTrace();}return null;
}public void download(String bucket, String objectId, OutputStream output) {//@formatter:offtry (InputStream input = minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {IOUtils.copyLarge(input, output);} catch (Exception e) {e.printStackTrace();}//@formatter:on
}

删除对象

public boolean deleteObject(String bucket, String objectId) {//@formatter:offtry {minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {e.printStackTrace();}//@formatter:onreturn true;
}

判断对象是否存在

public boolean objectExists(String bucket, String key) {//@formatter:offtry {// minio客户端未提供判断对象是否存在的方法,此方法中调用出现异常时说明对象不存在minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(key).build());} catch (Exception e) {return false;}//@formatter:onreturn true;
}

完整代码

import java.io.InputStream;
import java.io.OutputStream;
import java.net.MalformedURLException;
import java.net.URL;import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;import io.minio.BucketExistsArgs;
import io.minio.GetObjectArgs;
import io.minio.MakeBucketArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import io.minio.RemoveBucketArgs;
import io.minio.RemoveObjectArgs;
import io.minio.StatObjectArgs;public class S3MinioClientDemo {private static final Logger logger = LoggerFactory.getLogger(S3MinioClientDemo.class);private static final String HTTP_PROTOCOL = "http";private MinioClient minioClient;private String endpoint = "http://192.168.0.8:9200";private String accessKey = "testKey";private String secretKey = "testSecretKey";public void init() throws MalformedURLException {URL endpointUrl = new URL(endpoint);try {// url上无端口号时,识别http为80端口,https为443端口int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;//@formatter:offthis.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security).credentials(accessKey, secretKey).build();//@formatter:on// 忽略证书校验,防止自签名证书校验失败导致无法建立连接this.minioClient.ignoreCertCheck();} catch (Exception e) {e.printStackTrace();}}public boolean createBucket(String bucket) {try {boolean found = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());if (found) {logger.info("桶名[{}]已存在", bucket);return false;}minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());} catch (Exception e) {e.printStackTrace();}return true;}public boolean deleteBucket(String bucket) {try {minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());logger.info("删除桶[{}]成功", bucket);} catch (Exception e) {e.printStackTrace();return false;}return true;}public boolean bucketExists(String bucket) {try {return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());} catch (Exception e) {e.printStackTrace();return false;}}public void upload(String bucket, String objectId, InputStream input) {try {//@formatter:offminioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId).stream(input, input.available(), -1).build());//@formatter:on} catch (Exception e) {e.printStackTrace();}}public InputStream download(String bucket, String objectId) {try {return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {e.printStackTrace();}return null;}public void download(String bucket, String objectId, OutputStream output) {//@formatter:offtry (InputStream input = minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {IOUtils.copyLarge(input, output);} catch (Exception e) {e.printStackTrace();}//@formatter:on}public boolean objectExists(String bucket, String objectId) {//@formatter:offtry {// minio客户端未提供判断对象是否存在的方法,此方法中调用出现异常时说明对象不存在minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {return false;}//@formatter:onreturn true;}public boolean deleteObject(String bucket, String objectId) {//@formatter:offtry {minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucket).object(objectId).build());} catch (Exception e) {e.printStackTrace();}//@formatter:onreturn true;}public void close() {minioClient = null;}}

Amazon S3 SDK方式操作S3

官方API文档:https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

这里由于项目上提供的SDK和文档都是1.x的,这里就暂时只提供1.x的代码

引入依赖

Maven:

<dependency><groupId>com.amazonaws</groupId><artifactId>aws-java-sdk-s3</artifactId><version>1.11.300</version>
</dependency>

Gradle:

dependencies {   implementation 'com.amazonaws:aws-java-sdk-s3:1.11.300'
}

初始化客户端

private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class);private AmazonS3 s3client;
private String endpoint = "http://192.168.0.8:9200";
private String accessKey = "testKey";
private String secretKey = "testSecretKey";public void init() throws MalformedURLException {URL endpointUrl = new URL(endpoint);String protocol = endpointUrl.getProtocol();int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();ClientConfiguration clientConfig = new ClientConfiguration();clientConfig.setSignerOverride("S3SignerType");clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));// 禁用证书检查,避免https自签证书校验失败System.setProperty("com.amazonaws.sdk.disableCertChecking", "true");// 屏蔽 AWS 的 MD5 校验,避免校验导致的下载抛出异常问题System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true");AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey);// 创建 S3Client 实例AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig);s3client.setEndpoint(endpointUrl.getHost() + ":" + port);s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());this.s3client = s3client;
}

建桶

public boolean createBucket(String bucket)  {String bucketName = parseBucketName(bucket);try {if (s3client.doesBucketExist(bucketName)) {logger.warn("bucket[{}]已存在", bucketName);return false;}s3client.createBucket(bucketName);} catch (Exception e) {e.printStackTrace();}return true;
}

删桶

public boolean deleteBucket(String bucket)  {try {s3client.deleteBucket(bucket);logger.info("删除bucket[{}]成功", bucket);} catch (Exception e) {e.printStackTrace();return false;}return true;
}

判断桶是否存在

public boolean bucketExists(String bucket)  {try {return s3client.doesBucketExist(bucket);} catch (Exception e) {e.printStackTrace();}return false;
}

上传对象

public void upload(String bucket, String objectId, InputStream input)  {try {// 创建文件上传的元数据ObjectMetadata meta = new ObjectMetadata();// 设置文件上传长度meta.setContentLength(input.available());// 上传s3client.putObject(bucket, objectId, input, meta);} catch (Exception e) {e.printStackTrace();}
}

下载对象

public InputStream download(String bucket, String objectId)  {try {S3Object o = s3client.getObject(bucket, objectId);return o.getObjectContent();} catch (Exception e) {e.printStackTrace();}return null;
}public void download(String bucket, String objectId, OutputStream out)  {S3Object o = s3client.getObject(bucket, objectId);try (InputStream in = o.getObjectContent()) {IOUtils.copyLarge(in, out);} catch (Exception e) {e.printStackTrace();}
}

删除对象

public boolean deleteObject(String bucket, String objectId)  {try {s3client.deleteObject(bucket, objectId);} catch (Exception e) {e.printStackTrace();return false;}return true;
}

判断对象是否存在

public boolean existObject(String bucket, String objectId)  {try {return s3client.doesObjectExist(bucket, objectId);} catch (Exception e) {e.printStackTrace();return false;}
}

完整代码

import java.io.InputStream;
import java.io.OutputStream;
import java.net.MalformedURLException;
import java.net.URL;import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.S3ClientOptions;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.S3Object;/*** S3对象存储官方SDK实现** @author ZhangChenguang* @date 2023年2月2日*/
@SuppressWarnings("deprecation")
public class S3SdkDemo {private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class);private AmazonS3 s3client;private String endpoint = "http://192.168.0.8:9200";private String accessKey = "testKey";private String secretKey = "testSecretKey";public void init() throws MalformedURLException {URL endpointUrl = new URL(endpoint);String protocol = endpointUrl.getProtocol();int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();ClientConfiguration clientConfig = new ClientConfiguration();clientConfig.setSignerOverride("S3SignerType");clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));// 禁用证书检查,避免https自签证书校验失败System.setProperty("com.amazonaws.sdk.disableCertChecking", "true");// 屏蔽 AWS 的 MD5 校验,避免校验导致的下载抛出异常问题System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true");AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey);// 创建 S3Client 实例AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig);s3client.setEndpoint(endpointUrl.getHost() + ":" + port);s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());this.s3client = s3client;}public boolean createBucket(String bucket)  {try {s3client.createBucket(bucket);} catch (Exception e) {e.printStackTrace();}return true;}public boolean deleteBucket(String bucket)  {try {s3client.deleteBucket(bucket);logger.info("删除bucket[{}]成功", bucket);} catch (Exception e) {e.printStackTrace();return false;}return true;}public boolean bucketExists(String bucket)  {try {return s3client.doesBucketExist(bucket);} catch (Exception e) {e.printStackTrace();}return false;}public void upload(String bucket, String objectId, InputStream input)  {try {// 创建文件上传的元数据ObjectMetadata meta = new ObjectMetadata();// 设置文件上传长度meta.setContentLength(input.available());// 上传s3client.putObject(bucket, objectId, input, meta);} catch (Exception e) {e.printStackTrace();}}public InputStream download(String bucket, String objectId)  {try {S3Object o = s3client.getObject(bucket, objectId);return o.getObjectContent();} catch (Exception e) {e.printStackTrace();}return null;}public void download(String bucket, String objectId, OutputStream out)  {S3Object o = s3client.getObject(bucket, objectId);try (InputStream in = o.getObjectContent()) {IOUtils.copyLarge(in, out);} catch (Exception e) {e.printStackTrace();}}public boolean existObject(String bucket, String objectId)  {try {return s3client.doesObjectExist(bucket, objectId);} catch (Exception e) {e.printStackTrace();return false;}}public boolean deleteObject(String bucket, String objectId)  {try {s3client.deleteObject(bucket, objectId);} catch (Exception e) {e.printStackTrace();return false;}return true;}public void close()  {s3client = null;}
}

遇到的问题

1、bucket名称必须是小写,不支持下划线

  • 处理方式:写方法转换下bucket名称,将大写转小写,将下划线替换为中划线。

2、minio客户端下载非官方S3存储的文件时,如果响应头的Content-Length与实际文件大小不符,会导致minio客户端包装的okhttp3报错

报错信息:

Caused by: java.net.ProtocolException: unexpected end of streamat okhttp3.internal.http1.Http1ExchangeCodec$FixedLengthSource.read(Http1ExchangeCodec.java:430) ~[okhttp-3.14.9.jar:?]at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) ~[okhttp-3.14.9.jar:?]at okio.RealBufferedSource$1.read(RealBufferedSource.java:447) ~[okio-1.17.2.jar:?]at com.jiuqi.nr.file.utils.FileUtils.writeInput2Output(FileUtils.java:83) ~[nr.file-2.5.7.jar:?]at com.jiuqi.nr.file.impl.FileAreaServiceImpl.download(FileAreaServiceImpl.java:395) ~[nr.file-2.5.7.jar:?]... 122 more

抓包发现问题的图:

最终换成了S3官方SDK可用了。

PS:客户现场部署的S3是浪潮公司提供的,如果现场遇到这个情况,就不要固执去找对方对线了,完全没用。。

总结

S3存储的基本操作就记录到这里了,由于没有S3存储就没尝试官方SDK的V2版本,由于这些代码是总结时从业务代码里抽取出来的,可能会有点问题,但大体思路已经有了。

希望对读者有所用处,觉得写得不错和有帮到你,欢迎点个赞,您的支持就是我的鼓励!

http://www.hkea.cn/news/227033/

相关文章:

  • 怎么样做淘宝优惠券网站搜索引擎营销的优缺点
  • wordpress动态订单seo社区
  • 网站域没到期不能续费吗google谷歌搜索
  • 厦门好的做网站公司网络营销推广方式都有哪些
  • 重庆市建设工程信息官网站自己做网站的流程
  • 网站建设公司怎么做网络营销网站推广
  • 360应用商店seo服务套餐
  • 废橡胶网站建设个人博客网页设计
  • 什么网站做一手项目好域名查询官网
  • 做日用品的要找什么网站好站长工具端口检测
  • 贵州软件开发 网站开发手机版百度一下
  • 企业网站建立答辩问题百度怎么发布广告
  • 温州快建网站地推拉新接单网
  • 濉溪县城乡建设委员会燃气办网站热狗网站排名优化外包
  • 网站能不能自己做免费的seo教程
  • 湖南的商城网站建设优化教程网下载
  • 做网站需要哪些工程师西安seo诊断
  • tp做的网站封装成app2023北京封控了
  • 增城做网站要多少钱推广普通话手抄报
  • 石家庄网站系统开发智能搜索引擎
  • 迅速网站网络营销平台推广方案
  • 学前端要逛那些网站微信引流主动被加软件
  • 韩国flash网站免费手机网站建站平台
  • 东莞做网站卓诚网络昆明长尾词seo怎么优化
  • WordPress个性萌化插件郑州seo优化哪家好
  • 专业手机移动网站建设免费的seo优化
  • 西安网站建设王永杰域名注册 万网
  • 网站营销优化方案北京做的好的seo公司
  • 企业网站排名提升软件优化南宁seo优化
  • 创意合肥网站建设杭州seo公司排名