大数据与流处理 / 系统课时

双流匹配

双流匹配

双流匹配用于关联来自不同系统、到达顺序不确定的业务事件。本课以订单流和支付流为例,先按订单号分区,再用 KeyedCoProcessFunction 保存暂未匹配的数据,并通过事件时间定时器输出超时结果。

学习目标

  • 理解 connect、双侧 keyBy 与双输入处理函数的职责
  • 使用键控状态兼容“订单先到”和“支付先到”两种顺序
  • 使用事件时间定时器清理状态并识别未匹配事件

订单与支付的匹配模型

订单和支付来自两个独立数据源,不能假设哪一侧先到。两条流必须使用相同业务键进行 keyBy,再通过 connect 进入同一个键控双输入算子。当前键只保存自己的订单、支付和超时定时器,匹配成功后立即输出并清理状态。

CoProcessFunction 提供两个输入处理入口;需要键控状态和按键定时器时,应使用它的键控形式 KeyedCoProcessFunction。定时器负责给暂存状态设置生命周期,否则长期未匹配的数据会持续占用状态后端。

订单支付双流匹配时序图

实现订单支付双流匹配

示例使用三元组表示事件:订单为 (订单号, 金额, 事件时间),支付为 (订单号, 渠道, 事件时间)。等待时间设为 5 秒;有配对事件就输出 MATCHED,只有一侧到达则在水位线越过截止时间后输出 TIMEOUT。

java
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.OpenContext;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.api.common.typeinfo.TypeHint;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.streaming.api.TimerService;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction;
import org.apache.flink.util.Collector;

public class TwoStreamMatchDemo {

    private static final long WAIT_MILLIS = 5_000L;

    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
                StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);

        DataStream<Tuple3<String, Long, Long>> orders =
                env.fromElements(
                        Tuple3.of("O-100", 8_800L, 1_000L),
                        Tuple3.of("O-101", 5_600L, 2_000L)
                ).assignTimestampsAndWatermarks(
                        WatermarkStrategy
                                .<Tuple3<String, Long, Long>>forMonotonousTimestamps()
                                .withTimestampAssigner((event, previous) -> event.f2)
                );

        DataStream<Tuple3<String, String, Long>> payments =
                env.fromElements(
                        Tuple3.of("O-100", "CARD", 1_500L)
                ).assignTimestampsAndWatermarks(
                        WatermarkStrategy
                                .<Tuple3<String, String, Long>>forMonotonousTimestamps()
                                .withTimestampAssigner((event, previous) -> event.f2)
                );

        orders.keyBy(order -> order.f0)
                .connect(payments.keyBy(payment -> payment.f0))
                .process(new OrderPaymentMatcher())
                .print("result");

        env.execute("two-stream-match-demo");
    }

    private static class OrderPaymentMatcher extends
            KeyedCoProcessFunction<
                    String,
                    Tuple3<String, Long, Long>,
                    Tuple3<String, String, Long>,
                    String> {

        private transient ValueState<Tuple3<String, Long, Long>> orderState;
        private transient ValueState<Tuple3<String, String, Long>> paymentState;
        private transient ValueState<Long> timerState;

        @Override
        public void open(OpenContext openContext) {
            orderState = getRuntimeContext().getState(
                    new ValueStateDescriptor<>(
                            "order-state",
                            TypeInformation.of(
                                    new TypeHint<Tuple3<String, Long, Long>>() {})));
            paymentState = getRuntimeContext().getState(
                    new ValueStateDescriptor<>(
                            "payment-state",
                            TypeInformation.of(
                                    new TypeHint<Tuple3<String, String, Long>>() {})));
            timerState = getRuntimeContext().getState(
                    new ValueStateDescriptor<>("timer-state", Long.class));
        }

        @Override
        public void processElement1(
                Tuple3<String, Long, Long> order,
                Context ctx,
                Collector<String> out) throws Exception {

            Tuple3<String, String, Long> payment = paymentState.value();
            if (payment != null) {
                out.collect("MATCHED order=" + order.f0
                        + " amount=" + order.f1
                        + " channel=" + payment.f1);
                clearAll(ctx.timerService());
                return;
            }

            orderState.update(order);
            replaceTimer(order.f2 + WAIT_MILLIS, ctx.timerService());
        }

        @Override
        public void processElement2(
                Tuple3<String, String, Long> payment,
                Context ctx,
                Collector<String> out) throws Exception {

            Tuple3<String, Long, Long> order = orderState.value();
            if (order != null) {
                out.collect("MATCHED order=" + order.f0
                        + " amount=" + order.f1
                        + " channel=" + payment.f1);
                clearAll(ctx.timerService());
                return;
            }

            paymentState.update(payment);
            replaceTimer(payment.f2 + WAIT_MILLIS, ctx.timerService());
        }

        @Override
        public void onTimer(
                long timestamp,
                OnTimerContext ctx,
                Collector<String> out) throws Exception {

            Tuple3<String, Long, Long> order = orderState.value();
            Tuple3<String, String, Long> payment = paymentState.value();

            if (order != null) {
                out.collect("TIMEOUT order=" + order.f0);
            } else if (payment != null) {
                out.collect("TIMEOUT payment=" + payment.f0);
            }
            clearAll(ctx.timerService());
        }

        private void replaceTimer(
                long timestamp,
                TimerService timerService) throws Exception {

            Long oldTimer = timerState.value();
            if (oldTimer != null) {
                timerService.deleteEventTimeTimer(oldTimer);
            }
            timerService.registerEventTimeTimer(timestamp);
            timerState.update(timestamp);
        }

        private void clearAll(TimerService timerService) throws Exception {
            Long timer = timerState.value();
            if (timer != null) {
                timerService.deleteEventTimeTimer(timer);
            }
            orderState.clear();
            paymentState.clear();
            timerState.clear();
        }
    }
}

预期输出如下,<ID> 表示并行子任务编号。两条输入流的实际消费先后不固定,但状态逻辑会得到相同的业务结果。

text
result:<ID>> MATCHED order=O-100 amount=8800 channel=CARD
result:<ID>> TIMEOUT order=O-101

匹配与超时规则

收到任意一侧事件时,先查询另一侧状态。存在配对事件就输出并清理全部状态;不存在则保存当前事件,并注册“事件时间加等待时长”的定时器。示例使用有界数据源,输入结束时会推进最终水位线,因此未支付订单会触发超时。

生产任务应根据业务语义选择事件时间或处理时间。事件时间适合按事件发生时刻判断,但依赖正确的时间戳与 Watermark;处理时间实现简单,却会受机器负载、故障恢复和背压影响。无论选择哪一种,都要确保注册与删除使用同一种时间域。

生产边界与验收

重复订单、重复支付和状态覆盖会造成重复匹配或错误金额,通常需要事件唯一标识、去重状态或幂等下游。等待时长过短会把正常迟到事件判为超时,过长则增加状态规模和结果延迟,应结合业务延迟分布设置并监控超时率。

验收时至少覆盖订单先到、支付先到、仅订单、仅支付和重复事件五类数据。还应启用 Checkpoint,验证任务故障恢复后暂存状态与定时器仍能继续工作,而不是重新产生已输出结果。

按自己的节奏,确认真正理解。

目录与阅读设置

阅读设置

17

记录一个想法

双流匹配

0 / 10000 · 笔记保存到账号。未提交的修改会在关闭、离开或刷新后丢失。

登录哪都通