大数据与流处理 / 系统课时

清洗交易事件

清洗交易事件

本课实现一条可运行的 Flink 清洗链路:原始 JSON 先解析,再校验版本、类型、时间和交易字段;合法事件进入主流,异常事件携带失败原因进入侧输出流,供后续写入异常主题。

学习目标

  • 使用 ProcessFunction 完成解析、校验和标准化
  • 使用 OutputTag 把异常事件输出到独立侧流
  • 通过本地样例验证合法、业务无效和格式错误三类输入

明确清洗结果

清洗不是简单删除坏数据。主流只保留满足当前协议的标准事件;侧流保存失败原因和截断后的原始消息,便于审计、修复和重放。两条流必须由同一个算子产生,避免重复解析和两套校验规则逐渐不一致。

Flink 交易事件清洗与侧输出流程图

校验顺序应先处理结构错误,再处理业务约束。这样可以避免对不存在的节点取值,也能让异常原因更准确。生产环境还应附带 Kafka topic、partition、offset 等元数据,本地示例先聚焦清洗逻辑。

实现清洗作业

下面的 Maven 配置使用 Flink 2.3.0、Java 17 和 Jackson。将 Java 文件放到 src/main/java/com/anystack/flink/TransactionCleaningJob.java。

xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.anystack</groupId>
  <artifactId>flink-transaction-cleaning</artifactId>
  <version>1.0.0</version>

  <properties>
    <maven.compiler.release>17</maven.compiler.release>
    <flink.version>2.3.0</flink.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-java</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-clients</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.18.4</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.5.1</version>
      </plugin>
    </plugins>
  </build>
</project>

该配置在本地 JVM 中运行示例。提交到已有 Flink 集群时,通常应把集群已提供的 Flink 依赖改为 provided,避免重复打包运行时组件。

示例构造三条输入:一条合法事件、一条金额无效事件和一条损坏 JSON。OutputTag 使用匿名内部类保留泛型信息,ProcessFunction 在一次处理内决定写主流还是异常侧流。

java
package com.anystack.flink;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;

import java.util.Locale;
import java.util.Set;

public class TransactionCleaningJob {

    private static final OutputTag<String> INVALID_TAG =
            new OutputTag<String>("invalid-events") {};

    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
                StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);

        DataStreamSource<String> source = env.fromData(
                "{\"schemaVersion\":\"1.0\",\"eventId\":\"evt-1001\","
                        + "\"eventType\":\"PAYMENT_SUCCEEDED\",\"eventTime\":1767225600123,"
                        + "\"source\":\"trade-service\",\"traceId\":\"trace-9001\","
                        + "\"data\":{\"transactionId\":\"tx-1\",\"merchantId\":\"m-100\","
                        + "\"orderId\":\"order-1\",\"amountMinor\":12800,\"currency\":\"cny\"}}",
                "{\"schemaVersion\":\"1.0\",\"eventId\":\"evt-1002\","
                        + "\"eventType\":\"PAYMENT_SUCCEEDED\",\"eventTime\":1767225601123,"
                        + "\"source\":\"trade-service\",\"data\":{\"transactionId\":\"tx-2\","
                        + "\"merchantId\":\"m-100\",\"orderId\":\"order-2\","
                        + "\"amountMinor\":0,\"currency\":\"CNY\"}}",
                "{bad json}"
        );

        SingleOutputStreamOperator<String> cleaned =
                source.process(new CleanFunction());

        cleaned.print("clean");
        cleaned.getSideOutput(INVALID_TAG).print("invalid");

        env.execute("transaction-event-cleaning");
    }

    static final class CleanFunction extends ProcessFunction<String, String> {

        private static final Set<String> ALLOWED_TYPES = Set.of(
                "PAYMENT_CREATED",
                "PAYMENT_SUCCEEDED",
                "PAYMENT_FAILED",
                "REFUND_SUCCEEDED"
        );

        private transient ObjectMapper mapper;

        @Override
        public void processElement(
                String raw,
                Context ctx,
                Collector<String> out) throws Exception {

            ObjectMapper objectMapper = mapper();

            try {
                JsonNode root = objectMapper.readTree(raw);
                String reason = validate(root);

                if (reason != null) {
                    ctx.output(INVALID_TAG, errorJson(objectMapper, reason, raw));
                    return;
                }

                JsonNode data = root.path("data");
                ObjectNode clean = objectMapper.createObjectNode();
                clean.put("schemaVersion", text(root, "schemaVersion"));
                clean.put("eventId", text(root, "eventId"));
                clean.put("eventType", text(root, "eventType"));
                clean.put("eventTime", root.path("eventTime").asLong());
                clean.put("source", text(root, "source"));
                clean.put("traceId", text(root, "traceId"));

                ObjectNode cleanData = clean.putObject("data");
                cleanData.put("transactionId", text(data, "transactionId"));
                cleanData.put("merchantId", text(data, "merchantId"));
                cleanData.put("orderId", text(data, "orderId"));
                cleanData.put("amountMinor", data.path("amountMinor").asLong());
                cleanData.put(
                        "currency",
                        text(data, "currency").toUpperCase(Locale.ROOT)
                );

                out.collect(objectMapper.writeValueAsString(clean));
            } catch (Exception ex) {
                ctx.output(
                        INVALID_TAG,
                        errorJson(objectMapper, "JSON 解析失败", raw)
                );
            }
        }

        private String validate(JsonNode root) {
            if (root == null || !root.isObject()) {
                return "根节点必须是 JSON 对象";
            }
            if (!text(root, "schemaVersion").startsWith("1.")) {
                return "不支持的 schemaVersion";
            }
            if (text(root, "eventId").isBlank()) {
                return "eventId 不能为空";
            }
            if (!ALLOWED_TYPES.contains(text(root, "eventType"))) {
                return "eventType 不受支持";
            }
            if (root.path("eventTime").asLong(0) <= 0) {
                return "eventTime 必须是正整数毫秒";
            }
            if (text(root, "source").isBlank()) {
                return "source 不能为空";
            }

            JsonNode data = root.path("data");
            if (!data.isObject()) {
                return "data 必须是 JSON 对象";
            }
            if (text(data, "transactionId").isBlank()) {
                return "transactionId 不能为空";
            }
            if (text(data, "merchantId").isBlank()) {
                return "merchantId 不能为空";
            }
            if (text(data, "orderId").isBlank()) {
                return "orderId 不能为空";
            }
            if (data.path("amountMinor").asLong(0) <= 0) {
                return "amountMinor 必须大于 0";
            }

            String currency = text(data, "currency")
                    .toUpperCase(Locale.ROOT);
            if (!currency.matches("[A-Z]{3}")) {
                return "currency 必须是三位字母";
            }
            return null;
        }

        private ObjectMapper mapper() {
            if (mapper == null) {
                mapper = new ObjectMapper();
            }
            return mapper;
        }

        private static String text(JsonNode node, String field) {
            return node.path(field).asText("").trim();
        }

        private static String errorJson(
                ObjectMapper mapper,
                String reason,
                String raw) throws Exception {

            ObjectNode error = mapper.createObjectNode();
            error.put("reason", reason);
            error.put("raw", truncate(raw, 300));
            return mapper.writeValueAsString(error);
        }

        private static String truncate(String value, int maxLength) {
            return value.length() <= maxLength
                    ? value
                    : value.substring(0, maxLength);
        }
    }
}

mapper 使用 transient 并在 TaskManager 首次处理数据时初始化,避免把运行期对象状态当作作业配置序列化。异常侧流的类型是 String,必须使用同一个 INVALID_TAG 取回。

运行与生产边界

在包含 pom.xml 的目录执行:

bash
mvn -q compile exec:java \
  -Dexec.mainClass=com.anystack.flink.TransactionCleaningJob

预期输出如下,Flink 启停日志可能同时出现;主流与侧流由不同 sink 打印,三行先后次序可能变化,字段顺序以示例中的 Jackson 写入顺序为准。

text
clean> {"schemaVersion":"1.0","eventId":"evt-1001","eventType":"PAYMENT_SUCCEEDED","eventTime":1767225600123,"source":"trade-service","traceId":"trace-9001","data":{"transactionId":"tx-1","merchantId":"m-100","orderId":"order-1","amountMinor":12800,"currency":"CNY"}}
invalid> {"reason":"amountMinor 必须大于 0","raw":"{\"schemaVersion\":\"1.0\",\"eventId\":\"evt-1002\",\"eventType\":\"PAYMENT_SUCCEEDED\",\"eventTime\":1767225601123,\"source\":\"trade-service\",\"data\":{\"transactionId\":\"tx-2\",\"merchantId\":\"m-100\",\"orderId\":\"order-2\",\"amountMinor\":0,\"currency\":\"CNY\"}}"}
invalid> {"reason":"JSON 解析失败","raw":"{bad json}"}

生产接入时,把 env.fromData(...) 替换为 Kafka Source,把主流写入 trade-events-clean-v1,把侧流写入 trade-events-dlq-v1。异常记录还应补充原 topic、partition、offset 和消费时间,防止只能看到坏数据却无法定位来源。

同一个 eventId 重复到达不属于字段清洗错误,应在后续按键状态或外部幂等存储处理。事件时间过早或过晚也不应直接等同于 JSON 无效,应根据业务允许的时间范围单独标记。

异常消息可能包含敏感字段,写入侧流前应脱敏并限制长度。不要把完整堆栈写进每条错误事件,否则会放大消息体并泄漏实现细节;详细堆栈应进入作业日志和监控系统。

按自己的节奏,确认真正理解。

目录与阅读设置

阅读设置

17

记录一个想法

清洗交易事件

0 / 10000 · 笔记保存到账号。未提交的修改会在关闭、离开或刷新后丢失。

登录哪都通