大数据与流处理 / 系统课时
双流匹配
双流匹配
双流匹配用于关联来自不同系统、到达顺序不确定的业务事件。本课以订单流和支付流为例,先按订单号分区,再用 KeyedCoProcessFunction 保存暂未匹配的数据,并通过事件时间定时器输出超时结果。
学习目标
- 理解
connect、双侧keyBy与双输入处理函数的职责 - 使用键控状态兼容“订单先到”和“支付先到”两种顺序
- 使用事件时间定时器清理状态并识别未匹配事件
订单与支付的匹配模型
订单和支付来自两个独立数据源,不能假设哪一侧先到。两条流必须使用相同业务键进行 keyBy,再通过 connect 进入同一个键控双输入算子。当前键只保存自己的订单、支付和超时定时器,匹配成功后立即输出并清理状态。
CoProcessFunction 提供两个输入处理入口;需要键控状态和按键定时器时,应使用它的键控形式 KeyedCoProcessFunction。定时器负责给暂存状态设置生命周期,否则长期未匹配的数据会持续占用状态后端。

实现订单支付双流匹配
示例使用三元组表示事件:订单为 (订单号, 金额, 事件时间),支付为 (订单号, 渠道, 事件时间)。等待时间设为 5 秒;有配对事件就输出 MATCHED,只有一侧到达则在水位线越过截止时间后输出 TIMEOUT。
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.OpenContext;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.api.common.typeinfo.TypeHint;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.streaming.api.TimerService;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction;
import org.apache.flink.util.Collector;
public class TwoStreamMatchDemo {
private static final long WAIT_MILLIS = 5_000L;
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
DataStream<Tuple3<String, Long, Long>> orders =
env.fromElements(
Tuple3.of("O-100", 8_800L, 1_000L),
Tuple3.of("O-101", 5_600L, 2_000L)
).assignTimestampsAndWatermarks(
WatermarkStrategy
.<Tuple3<String, Long, Long>>forMonotonousTimestamps()
.withTimestampAssigner((event, previous) -> event.f2)
);
DataStream<Tuple3<String, String, Long>> payments =
env.fromElements(
Tuple3.of("O-100", "CARD", 1_500L)
).assignTimestampsAndWatermarks(
WatermarkStrategy
.<Tuple3<String, String, Long>>forMonotonousTimestamps()
.withTimestampAssigner((event, previous) -> event.f2)
);
orders.keyBy(order -> order.f0)
.connect(payments.keyBy(payment -> payment.f0))
.process(new OrderPaymentMatcher())
.print("result");
env.execute("two-stream-match-demo");
}
private static class OrderPaymentMatcher extends
KeyedCoProcessFunction<
String,
Tuple3<String, Long, Long>,
Tuple3<String, String, Long>,
String> {
private transient ValueState<Tuple3<String, Long, Long>> orderState;
private transient ValueState<Tuple3<String, String, Long>> paymentState;
private transient ValueState<Long> timerState;
@Override
public void open(OpenContext openContext) {
orderState = getRuntimeContext().getState(
new ValueStateDescriptor<>(
"order-state",
TypeInformation.of(
new TypeHint<Tuple3<String, Long, Long>>() {})));
paymentState = getRuntimeContext().getState(
new ValueStateDescriptor<>(
"payment-state",
TypeInformation.of(
new TypeHint<Tuple3<String, String, Long>>() {})));
timerState = getRuntimeContext().getState(
new ValueStateDescriptor<>("timer-state", Long.class));
}
@Override
public void processElement1(
Tuple3<String, Long, Long> order,
Context ctx,
Collector<String> out) throws Exception {
Tuple3<String, String, Long> payment = paymentState.value();
if (payment != null) {
out.collect("MATCHED order=" + order.f0
+ " amount=" + order.f1
+ " channel=" + payment.f1);
clearAll(ctx.timerService());
return;
}
orderState.update(order);
replaceTimer(order.f2 + WAIT_MILLIS, ctx.timerService());
}
@Override
public void processElement2(
Tuple3<String, String, Long> payment,
Context ctx,
Collector<String> out) throws Exception {
Tuple3<String, Long, Long> order = orderState.value();
if (order != null) {
out.collect("MATCHED order=" + order.f0
+ " amount=" + order.f1
+ " channel=" + payment.f1);
clearAll(ctx.timerService());
return;
}
paymentState.update(payment);
replaceTimer(payment.f2 + WAIT_MILLIS, ctx.timerService());
}
@Override
public void onTimer(
long timestamp,
OnTimerContext ctx,
Collector<String> out) throws Exception {
Tuple3<String, Long, Long> order = orderState.value();
Tuple3<String, String, Long> payment = paymentState.value();
if (order != null) {
out.collect("TIMEOUT order=" + order.f0);
} else if (payment != null) {
out.collect("TIMEOUT payment=" + payment.f0);
}
clearAll(ctx.timerService());
}
private void replaceTimer(
long timestamp,
TimerService timerService) throws Exception {
Long oldTimer = timerState.value();
if (oldTimer != null) {
timerService.deleteEventTimeTimer(oldTimer);
}
timerService.registerEventTimeTimer(timestamp);
timerState.update(timestamp);
}
private void clearAll(TimerService timerService) throws Exception {
Long timer = timerState.value();
if (timer != null) {
timerService.deleteEventTimeTimer(timer);
}
orderState.clear();
paymentState.clear();
timerState.clear();
}
}
}预期输出如下,<ID> 表示并行子任务编号。两条输入流的实际消费先后不固定,但状态逻辑会得到相同的业务结果。
result:<ID>> MATCHED order=O-100 amount=8800 channel=CARD
result:<ID>> TIMEOUT order=O-101匹配与超时规则
收到任意一侧事件时,先查询另一侧状态。存在配对事件就输出并清理全部状态;不存在则保存当前事件,并注册“事件时间加等待时长”的定时器。示例使用有界数据源,输入结束时会推进最终水位线,因此未支付订单会触发超时。
生产任务应根据业务语义选择事件时间或处理时间。事件时间适合按事件发生时刻判断,但依赖正确的时间戳与 Watermark;处理时间实现简单,却会受机器负载、故障恢复和背压影响。无论选择哪一种,都要确保注册与删除使用同一种时间域。
生产边界与验收
重复订单、重复支付和状态覆盖会造成重复匹配或错误金额,通常需要事件唯一标识、去重状态或幂等下游。等待时长过短会把正常迟到事件判为超时,过长则增加状态规模和结果延迟,应结合业务延迟分布设置并监控超时率。
验收时至少覆盖订单先到、支付先到、仅订单、仅支付和重复事件五类数据。还应启用 Checkpoint,验证任务故障恢复后暂存状态与定时器仍能继续工作,而不是重新产生已输出结果。