大数据与流处理 / 系统课时
清洗交易事件
清洗交易事件
本课实现一条可运行的 Flink 清洗链路:原始 JSON 先解析,再校验版本、类型、时间和交易字段;合法事件进入主流,异常事件携带失败原因进入侧输出流,供后续写入异常主题。
学习目标
- 使用
ProcessFunction完成解析、校验和标准化 - 使用
OutputTag把异常事件输出到独立侧流 - 通过本地样例验证合法、业务无效和格式错误三类输入
明确清洗结果
清洗不是简单删除坏数据。主流只保留满足当前协议的标准事件;侧流保存失败原因和截断后的原始消息,便于审计、修复和重放。两条流必须由同一个算子产生,避免重复解析和两套校验规则逐渐不一致。

校验顺序应先处理结构错误,再处理业务约束。这样可以避免对不存在的节点取值,也能让异常原因更准确。生产环境还应附带 Kafka topic、partition、offset 等元数据,本地示例先聚焦清洗逻辑。
实现清洗作业
下面的 Maven 配置使用 Flink 2.3.0、Java 17 和 Jackson。将 Java 文件放到 src/main/java/com/anystack/flink/TransactionCleaningJob.java。
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.anystack</groupId>
<artifactId>flink-transaction-cleaning</artifactId>
<version>1.0.0</version>
<properties>
<maven.compiler.release>17</maven.compiler.release>
<flink.version>2.3.0</flink.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.18.4</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.5.1</version>
</plugin>
</plugins>
</build>
</project>该配置在本地 JVM 中运行示例。提交到已有 Flink 集群时,通常应把集群已提供的 Flink 依赖改为 provided,避免重复打包运行时组件。
示例构造三条输入:一条合法事件、一条金额无效事件和一条损坏 JSON。OutputTag 使用匿名内部类保留泛型信息,ProcessFunction 在一次处理内决定写主流还是异常侧流。
package com.anystack.flink;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;
import java.util.Locale;
import java.util.Set;
public class TransactionCleaningJob {
private static final OutputTag<String> INVALID_TAG =
new OutputTag<String>("invalid-events") {};
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
DataStreamSource<String> source = env.fromData(
"{\"schemaVersion\":\"1.0\",\"eventId\":\"evt-1001\","
+ "\"eventType\":\"PAYMENT_SUCCEEDED\",\"eventTime\":1767225600123,"
+ "\"source\":\"trade-service\",\"traceId\":\"trace-9001\","
+ "\"data\":{\"transactionId\":\"tx-1\",\"merchantId\":\"m-100\","
+ "\"orderId\":\"order-1\",\"amountMinor\":12800,\"currency\":\"cny\"}}",
"{\"schemaVersion\":\"1.0\",\"eventId\":\"evt-1002\","
+ "\"eventType\":\"PAYMENT_SUCCEEDED\",\"eventTime\":1767225601123,"
+ "\"source\":\"trade-service\",\"data\":{\"transactionId\":\"tx-2\","
+ "\"merchantId\":\"m-100\",\"orderId\":\"order-2\","
+ "\"amountMinor\":0,\"currency\":\"CNY\"}}",
"{bad json}"
);
SingleOutputStreamOperator<String> cleaned =
source.process(new CleanFunction());
cleaned.print("clean");
cleaned.getSideOutput(INVALID_TAG).print("invalid");
env.execute("transaction-event-cleaning");
}
static final class CleanFunction extends ProcessFunction<String, String> {
private static final Set<String> ALLOWED_TYPES = Set.of(
"PAYMENT_CREATED",
"PAYMENT_SUCCEEDED",
"PAYMENT_FAILED",
"REFUND_SUCCEEDED"
);
private transient ObjectMapper mapper;
@Override
public void processElement(
String raw,
Context ctx,
Collector<String> out) throws Exception {
ObjectMapper objectMapper = mapper();
try {
JsonNode root = objectMapper.readTree(raw);
String reason = validate(root);
if (reason != null) {
ctx.output(INVALID_TAG, errorJson(objectMapper, reason, raw));
return;
}
JsonNode data = root.path("data");
ObjectNode clean = objectMapper.createObjectNode();
clean.put("schemaVersion", text(root, "schemaVersion"));
clean.put("eventId", text(root, "eventId"));
clean.put("eventType", text(root, "eventType"));
clean.put("eventTime", root.path("eventTime").asLong());
clean.put("source", text(root, "source"));
clean.put("traceId", text(root, "traceId"));
ObjectNode cleanData = clean.putObject("data");
cleanData.put("transactionId", text(data, "transactionId"));
cleanData.put("merchantId", text(data, "merchantId"));
cleanData.put("orderId", text(data, "orderId"));
cleanData.put("amountMinor", data.path("amountMinor").asLong());
cleanData.put(
"currency",
text(data, "currency").toUpperCase(Locale.ROOT)
);
out.collect(objectMapper.writeValueAsString(clean));
} catch (Exception ex) {
ctx.output(
INVALID_TAG,
errorJson(objectMapper, "JSON 解析失败", raw)
);
}
}
private String validate(JsonNode root) {
if (root == null || !root.isObject()) {
return "根节点必须是 JSON 对象";
}
if (!text(root, "schemaVersion").startsWith("1.")) {
return "不支持的 schemaVersion";
}
if (text(root, "eventId").isBlank()) {
return "eventId 不能为空";
}
if (!ALLOWED_TYPES.contains(text(root, "eventType"))) {
return "eventType 不受支持";
}
if (root.path("eventTime").asLong(0) <= 0) {
return "eventTime 必须是正整数毫秒";
}
if (text(root, "source").isBlank()) {
return "source 不能为空";
}
JsonNode data = root.path("data");
if (!data.isObject()) {
return "data 必须是 JSON 对象";
}
if (text(data, "transactionId").isBlank()) {
return "transactionId 不能为空";
}
if (text(data, "merchantId").isBlank()) {
return "merchantId 不能为空";
}
if (text(data, "orderId").isBlank()) {
return "orderId 不能为空";
}
if (data.path("amountMinor").asLong(0) <= 0) {
return "amountMinor 必须大于 0";
}
String currency = text(data, "currency")
.toUpperCase(Locale.ROOT);
if (!currency.matches("[A-Z]{3}")) {
return "currency 必须是三位字母";
}
return null;
}
private ObjectMapper mapper() {
if (mapper == null) {
mapper = new ObjectMapper();
}
return mapper;
}
private static String text(JsonNode node, String field) {
return node.path(field).asText("").trim();
}
private static String errorJson(
ObjectMapper mapper,
String reason,
String raw) throws Exception {
ObjectNode error = mapper.createObjectNode();
error.put("reason", reason);
error.put("raw", truncate(raw, 300));
return mapper.writeValueAsString(error);
}
private static String truncate(String value, int maxLength) {
return value.length() <= maxLength
? value
: value.substring(0, maxLength);
}
}
}mapper 使用 transient 并在 TaskManager 首次处理数据时初始化,避免把运行期对象状态当作作业配置序列化。异常侧流的类型是 String,必须使用同一个 INVALID_TAG 取回。
运行与生产边界
在包含 pom.xml 的目录执行:
mvn -q compile exec:java \
-Dexec.mainClass=com.anystack.flink.TransactionCleaningJob预期输出如下,Flink 启停日志可能同时出现;主流与侧流由不同 sink 打印,三行先后次序可能变化,字段顺序以示例中的 Jackson 写入顺序为准。
clean> {"schemaVersion":"1.0","eventId":"evt-1001","eventType":"PAYMENT_SUCCEEDED","eventTime":1767225600123,"source":"trade-service","traceId":"trace-9001","data":{"transactionId":"tx-1","merchantId":"m-100","orderId":"order-1","amountMinor":12800,"currency":"CNY"}}
invalid> {"reason":"amountMinor 必须大于 0","raw":"{\"schemaVersion\":\"1.0\",\"eventId\":\"evt-1002\",\"eventType\":\"PAYMENT_SUCCEEDED\",\"eventTime\":1767225601123,\"source\":\"trade-service\",\"data\":{\"transactionId\":\"tx-2\",\"merchantId\":\"m-100\",\"orderId\":\"order-2\",\"amountMinor\":0,\"currency\":\"CNY\"}}"}
invalid> {"reason":"JSON 解析失败","raw":"{bad json}"}生产接入时,把 env.fromData(...) 替换为 Kafka Source,把主流写入 trade-events-clean-v1,把侧流写入 trade-events-dlq-v1。异常记录还应补充原 topic、partition、offset 和消费时间,防止只能看到坏数据却无法定位来源。
同一个 eventId 重复到达不属于字段清洗错误,应在后续按键状态或外部幂等存储处理。事件时间过早或过晚也不应直接等同于 JSON 无效,应根据业务允许的时间范围单独标记。
异常消息可能包含敏感字段,写入侧流前应脱敏并限制长度。不要把完整堆栈写进每条错误事件,否则会放大消息体并泄漏实现细节;详细堆栈应进入作业日志和监控系统。