Java 后端 / 系统课时

接入对象存储

接入对象存储

对象存储适合保存图片、附件和导出文件,但应用不应把大文件长期经由自身转发。本课使用 S3 兼容接口实现服务端上传、预签名下载与元数据记录,并明确对象键、数据库记录和访问权限的边界。

学习目标

  • 配置 S3 兼容客户端并上传对象
  • 生成有限时效的预签名访问地址
  • 使用数据库元数据管理对象键、文件属性和业务归属

设计文件访问链路

对象存储负责保存二进制内容,数据库负责保存对象键、原始文件名、内容类型、大小和业务归属。公开访问地址不应永久写入数据库,因为域名、签名和有效期都可能变化;数据库应保存稳定的对象键,需要访问时再生成 URL。

对于小文件,应用可以接收后上传;对于大文件,更推荐由应用生成预签名上传地址,让客户端直接上传到对象存储。下面先实现容易验证的服务端上传与预签名下载。

对象上传、元数据和预签名访问数据流

添加依赖与连接配置

Maven 项目加入 AWS SDK for Java 2.x 的 S3 模块。版本号应由项目的依赖管理统一控制。

xml
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>bom</artifactId>
      <version>2.25.60</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-jdbc</artifactId>
  </dependency>
</dependencies>

配置保存端点、区域和桶名,访问密钥通过环境变量注入。S3 兼容服务常需要启用路径风格访问;真实 AWS S3 通常可以关闭该选项。

yaml
storage:
  endpoint: ${STORAGE_ENDPOINT}
  region: ${STORAGE_REGION:us-east-1}
  bucket: ${STORAGE_BUCKET}
  access-key: ${STORAGE_ACCESS_KEY}
  secret-key: ${STORAGE_SECRET_KEY}
  path-style-access: true

生效行为:应用启动时创建 S3Client 和 S3Presigner,后续请求复用这两个线程安全客户端。不要为每次上传重新创建连接对象。

创建客户端 Bean

下面配置适用于支持 S3 协议的对象存储。若使用 AWS 默认凭证链,可去掉静态凭证并改用 DefaultCredentialsProvider。

java
package com.example.storage;

import java.net.URI;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;

@ConfigurationProperties(prefix = "storage")
record StorageProperties(
        URI endpoint,
        String region,
        String bucket,
        String accessKey,
        String secretKey,
        boolean pathStyleAccess) {
}

@Configuration
@EnableConfigurationProperties(StorageProperties.class)
class StorageConfiguration {
    @Bean
    S3Client s3Client(StorageProperties properties) {
        return S3Client.builder()
                .endpointOverride(properties.endpoint())
                .region(Region.of(properties.region()))
                .credentialsProvider(credentials(properties))
                .serviceConfiguration(serviceConfiguration(properties))
                .build();
    }

    @Bean
    S3Presigner s3Presigner(StorageProperties properties) {
        return S3Presigner.builder()
                .endpointOverride(properties.endpoint())
                .region(Region.of(properties.region()))
                .credentialsProvider(credentials(properties))
                .serviceConfiguration(serviceConfiguration(properties))
                .build();
    }

    private StaticCredentialsProvider credentials(StorageProperties properties) {
        AwsBasicCredentials credentials = AwsBasicCredentials.create(
                properties.accessKey(), properties.secretKey());
        return StaticCredentialsProvider.create(credentials);
    }

    private S3Configuration serviceConfiguration(StorageProperties properties) {
        return S3Configuration.builder()
                .pathStyleAccessEnabled(properties.pathStyleAccess())
                .build();
    }
}

配置对象绑定成功后,端点、桶名和凭证会统一注入。启动失败时先检查环境变量是否存在,再确认端点 URI 是否包含 http:// 或 https://。

建立元数据表

元数据表保存稳定业务 ID 与对象键,不保存预签名 URL。object_key 唯一约束可防止不同业务记录误指向同一对象。

sql
CREATE TABLE file_object (
    id UUID PRIMARY KEY,
    object_key VARCHAR(500) NOT NULL UNIQUE,
    original_filename VARCHAR(255) NOT NULL,
    content_type VARCHAR(100) NOT NULL,
    size_bytes BIGINT NOT NULL,
    owner_id VARCHAR(100) NOT NULL,
    status VARCHAR(20) NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

预期结果:

text
CREATE TABLE

不同数据库的建表提示可能不同。生产环境应使用 Flyway 或 Liquibase 管理这段迁移,而不是在应用启动时重复执行。

实现上传、元数据和预签名访问

服务先校验文件并生成不可预测的对象键,再上传二进制并写入元数据。若数据库写入失败,会尝试删除已上传对象,避免产生无人引用的孤儿文件。

java
package com.example.storage;

import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
import java.util.Map;
import java.util.UUID;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;

@Service
public class ObjectStorageService {
    private final S3Client s3Client;
    private final S3Presigner presigner;
    private final StorageProperties properties;
    private final JdbcTemplate jdbc;

    public ObjectStorageService(
            S3Client s3Client,
            S3Presigner presigner,
            StorageProperties properties,
            JdbcTemplate jdbc) {
        this.s3Client = s3Client;
        this.presigner = presigner;
        this.properties = properties;
        this.jdbc = jdbc;
    }

    public StoredFile upload(MultipartFile file, String ownerId) throws IOException {
        if (file.isEmpty() || file.getSize() > 10 * 1024 * 1024) {
            throw new IllegalArgumentException("文件为空或超过 10 MiB");
        }

        UUID id = UUID.randomUUID();
        String objectKey = ownerId + "/" + id;
        String filename = file.getOriginalFilename() == null
                ? "unnamed"
                : file.getOriginalFilename();
        String contentType = file.getContentType() == null
                ? "application/octet-stream"
                : file.getContentType();

        PutObjectRequest request = PutObjectRequest.builder()
                .bucket(properties.bucket())
                .key(objectKey)
                .contentType(contentType)
                .metadata(Map.of("owner-id", ownerId))
                .build();

        try (var input = file.getInputStream()) {
            s3Client.putObject(
                    request,
                    RequestBody.fromInputStream(input, file.getSize()));
        }

        try {
            jdbc.update("""
                    INSERT INTO file_object
                    (id, object_key, original_filename, content_type,
                     size_bytes, owner_id, status, created_at)
                    VALUES (?, ?, ?, ?, ?, ?, ?, ?)
                    """,
                    id, objectKey, filename, contentType,
                    file.getSize(), ownerId, "READY", Instant.now());
        } catch (RuntimeException ex) {
            s3Client.deleteObject(DeleteObjectRequest.builder()
                    .bucket(properties.bucket())
                    .key(objectKey)
                    .build());
            throw ex;
        }

        return new StoredFile(
                id, objectKey, filename, contentType, file.getSize(), ownerId);
    }

    public String createDownloadUrl(UUID fileId, String ownerId, Duration ttl) {
        if (ttl.isNegative() || ttl.isZero()
                || ttl.compareTo(Duration.ofMinutes(15)) > 0) {
            throw new IllegalArgumentException(
                    "下载地址有效期必须在 1 秒到 15 分钟之间");
        }

        StoredFile file = jdbc.queryForObject("""
                SELECT id, object_key, original_filename,
                       content_type, size_bytes, owner_id
                FROM file_object
                WHERE id = ? AND status = 'READY'
                """,
                (rs, rowNum) -> new StoredFile(
                        rs.getObject("id", UUID.class),
                        rs.getString("object_key"),
                        rs.getString("original_filename"),
                        rs.getString("content_type"),
                        rs.getLong("size_bytes"),
                        rs.getString("owner_id")),
                fileId);

        if (file == null || !ownerId.equals(file.ownerId())) {
            throw new SecurityException("无权访问该文件");
        }

        GetObjectRequest getObject = GetObjectRequest.builder()
                .bucket(properties.bucket())
                .key(file.objectKey())
                .build();
        GetObjectPresignRequest request = GetObjectPresignRequest.builder()
                .signatureDuration(ttl)
                .getObjectRequest(getObject)
                .build();
        return presigner.presignGetObject(request).url().toString();
    }

    public record StoredFile(
            UUID id,
            String objectKey,
            String originalFilename,
            String contentType,
            long size,
            String ownerId) {
    }
}

这是可集成代码,依赖 Spring MVC、Spring JDBC 和已配置的数据源。对象上传与数据库写入无法组成同一个本地事务,因此删除失败时还应记录补偿任务,由后台作业再次清理。

暴露接口并验证

控制器接收文件并返回元数据。下载接口先按业务 ID 与所有者校验权限,再生成五分钟有效的 URL,客户端随后直接访问对象存储。

java
package com.example.storage;

import java.io.IOException;
import java.time.Duration;
import java.util.Map;
import java.util.UUID;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RequestPart;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

@RestController
class FileController {
    private final ObjectStorageService storageService;

    FileController(ObjectStorageService storageService) {
        this.storageService = storageService;
    }

    @PostMapping("/files")
    ObjectStorageService.StoredFile upload(
            @RequestPart MultipartFile file,
            @RequestParam String ownerId) throws IOException {
        return storageService.upload(file, ownerId);
    }

    @GetMapping("/files/{id}/url")
    Map<String, String> downloadUrl(
            @PathVariable UUID id,
            @RequestParam String ownerId) {
        String url = storageService.createDownloadUrl(
                id, ownerId, Duration.ofMinutes(5));
        return Map.of("url", url);
    }
}

使用下面命令上传一个本地文件:

bash
curl -F "file=@<PATH>/manual.pdf" \
  "http://localhost:<PORT>/files?ownerId=user-1001"

预期结果:

json
{
  "id": "<ID>",
  "objectKey": "user-1001/<ID>",
  "originalFilename": "manual.pdf",
  "contentType": "application/pdf",
  "size": 24576,
  "ownerId": "user-1001"
}

<PATH>、<PORT>、<ID> 和文件大小会随环境变化。再调用 /files/<ID>/url?ownerId=user-1001,预期返回包含临时签名参数的 URL;该地址在五分钟后失效。

安全与一致性边界

上传前应限制大小和允许的内容类型,必要时执行病毒扫描。不要相信客户端提供的扩展名,也不要把访问密钥返回给前端。预签名 URL 只授予单个对象和有限时间的权限,生成前仍需检查当前用户是否有权访问对应元数据记录。

删除文件时先把元数据状态改为 DELETING,再异步删除对象,成功后标记 DELETED。这样即使对象存储暂时不可用,也能通过状态和补偿任务继续处理,而不是让数据库记录与真实对象永久失配。

按自己的节奏,确认真正理解。

目录与阅读设置

阅读设置

17

记录一个想法

接入对象存储

0 / 10000 · 笔记保存到账号。未提交的修改会在关闭、离开或刷新后丢失。

登录哪都通