大数据与流处理 / 系统课时

检测交易异常

检测交易异常

本课把支付超时、短时高频和异常金额三类规则落成一个可运行的 Flink 作业。完成后,你能够为不同规则选择正确的分区键,使用键控状态保存上下文,并用事件时间定时器在条件满足时输出统一告警。

学习目标

  • 为支付超时和用户行为规则选择不同的 keyBy 键
  • 使用 ValueState 与事件时间定时器保存并清理检测上下文
  • 验证高频、超额和超时告警是否按预期产生

规则拆分与键选择

三类异常不能强行放进同一个键控算子。支付超时需要按 orderId 聚合创建与支付事件;短时高频需要按 userId 统计;异常金额虽然不依赖历史状态,但可与用户行为检测放在同一分支中输出。

规则 分区键 状态 定时器 触发条件
支付超时 orderId 待支付事件、超时时刻 事件时间 创建后 5 分钟仍未支付
短时高频 userId 窗口计数、窗口结束时间 事件时间 60 秒内支付达到 4 次
异常金额 userId 无必需状态 无 单笔支付金额超过 10000

交易异常检测的数据流与定时器关系

实现状态与定时器检测

下面的完整示例使用固定测试事件。项目需引入与集群版本一致的 Flink DataStream 依赖;示例时间戳是确定值,因此每次运行都能复现相同的三类告警。

java
package com.anystack.flink.trade;

import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.OpenContext;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
import org.apache.flink.util.Collector;

import java.io.Serializable;
import java.time.Duration;

public class TradeAnomalyJob {
    private static final long PAYMENT_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
    private static final long FREQUENCY_WINDOW_MS = Duration.ofSeconds(60).toMillis();
    private static final int FREQUENCY_THRESHOLD = 4;
    private static final double AMOUNT_THRESHOLD = 10_000D;

    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
                StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);

        long base = 1_700_000_000_000L;
        DataStream<TradeEvent> events = env.fromElements(
                new TradeEvent("O-1", "U-1", "CREATE", 100D, base),
                new TradeEvent("O-2", "U-2", "CREATE", 200D, base + 5_000),
                new TradeEvent("O-1", "U-1", "PAY", 100D, base + 10_000),
                new TradeEvent("O-3", "U-1", "PAY", 10D, base + 20_000),
                new TradeEvent("O-4", "U-1", "PAY", 20D, base + 30_000),
                new TradeEvent("O-5", "U-1", "PAY", 40D, base + 40_000),
                new TradeEvent("O-6", "U-3", "PAY", 30_000D, base + 50_000)
        ).assignTimestampsAndWatermarks(
                WatermarkStrategy.<TradeEvent>forBoundedOutOfOrderness(
                                Duration.ofSeconds(2))
                        .withTimestampAssigner((event, previous) -> event.eventTime)
        );

        DataStream<Alert> timeoutAlerts = events
                .keyBy(event -> event.orderId)
                .process(new PaymentTimeoutDetector());

        DataStream<Alert> behaviorAlerts = events
                .keyBy(event -> event.userId)
                .process(new BehaviorDetector());

        timeoutAlerts
                .union(behaviorAlerts)
                .map(Alert::toString)
                .print();

        env.execute("trade-anomaly-detection");
    }

    public static class PaymentTimeoutDetector
            extends KeyedProcessFunction<String, TradeEvent, Alert> {
        private transient ValueState<TradeEvent> pendingCreate;
        private transient ValueState<Long> timeoutAt;

        @Override
        public void open(OpenContext openContext) {
            pendingCreate = getRuntimeContext().getState(
                    new ValueStateDescriptor<>("pending-create", TradeEvent.class));
            timeoutAt = getRuntimeContext().getState(
                    new ValueStateDescriptor<>("timeout-at", Long.class));
        }

        @Override
        public void processElement(
                TradeEvent event,
                Context ctx,
                Collector<Alert> out) throws Exception {
            if ("CREATE".equals(event.eventType)) {
                Long oldTimer = timeoutAt.value();
                if (oldTimer != null) {
                    ctx.timerService().deleteEventTimeTimer(oldTimer);
                }

                long timer = event.eventTime + PAYMENT_TIMEOUT_MS;
                pendingCreate.update(event);
                timeoutAt.update(timer);
                ctx.timerService().registerEventTimeTimer(timer);
                return;
            }

            if ("PAY".equals(event.eventType) && pendingCreate.value() != null) {
                Long timer = timeoutAt.value();
                if (timer != null) {
                    ctx.timerService().deleteEventTimeTimer(timer);
                }
                pendingCreate.clear();
                timeoutAt.clear();
            }
        }

        @Override
        public void onTimer(
                long timestamp,
                OnTimerContext ctx,
                Collector<Alert> out) throws Exception {
            TradeEvent create = pendingCreate.value();
            Long expectedTimer = timeoutAt.value();

            if (create != null && expectedTimer != null && timestamp == expectedTimer) {
                out.collect(new Alert(
                        "PAYMENT_TIMEOUT",
                        create.orderId,
                        "创建后 5 分钟仍未支付"));
                pendingCreate.clear();
                timeoutAt.clear();
            }
        }
    }

    public static class BehaviorDetector
            extends KeyedProcessFunction<String, TradeEvent, Alert> {
        private transient ValueState<Integer> windowCount;
        private transient ValueState<Long> windowEnd;

        @Override
        public void open(OpenContext openContext) {
            windowCount = getRuntimeContext().getState(
                    new ValueStateDescriptor<>("window-count", Integer.class));
            windowEnd = getRuntimeContext().getState(
                    new ValueStateDescriptor<>("window-end", Long.class));
        }

        @Override
        public void processElement(
                TradeEvent event,
                Context ctx,
                Collector<Alert> out) throws Exception {
            if (!"PAY".equals(event.eventType)) {
                return;
            }

            if (event.amount > AMOUNT_THRESHOLD) {
                out.collect(new Alert(
                        "ABNORMAL_AMOUNT",
                        event.orderId,
                        "金额 " + event.amount + " 超过阈值 " + AMOUNT_THRESHOLD));
            }

            Long end = windowEnd.value();
            if (end == null || event.eventTime >= end) {
                if (end != null) {
                    ctx.timerService().deleteEventTimeTimer(end);
                }

                long newEnd = event.eventTime + FREQUENCY_WINDOW_MS;
                windowCount.update(1);
                windowEnd.update(newEnd);
                ctx.timerService().registerEventTimeTimer(newEnd);
                return;
            }

            Integer current = windowCount.value();
            int next = (current == null ? 0 : current) + 1;
            windowCount.update(next);

            if (next == FREQUENCY_THRESHOLD) {
                out.collect(new Alert(
                        "HIGH_FREQUENCY",
                        event.userId,
                        "60 秒内支付次数达到 " + next));
            }
        }

        @Override
        public void onTimer(
                long timestamp,
                OnTimerContext ctx,
                Collector<Alert> out) throws Exception {
            Long end = windowEnd.value();
            if (end != null && timestamp == end) {
                windowCount.clear();
                windowEnd.clear();
            }
        }
    }

    public static class TradeEvent implements Serializable {
        public String orderId;
        public String userId;
        public String eventType;
        public double amount;
        public long eventTime;

        public TradeEvent() {
        }

        public TradeEvent(
                String orderId,
                String userId,
                String eventType,
                double amount,
                long eventTime) {
            this.orderId = orderId;
            this.userId = userId;
            this.eventType = eventType;
            this.amount = amount;
            this.eventTime = eventTime;
        }
    }

    public static class Alert implements Serializable {
        public String type;
        public String key;
        public String message;

        public Alert() {
        }

        public Alert(String type, String key, String message) {
            this.type = type;
            this.key = key;
            this.message = message;
        }

        @Override
        public String toString() {
            return type + " | key=" + key + " | " + message;
        }
    }
}

预期输出如下,两个分支在 union 后的行顺序可能不同。

text
HIGH_FREQUENCY | key=U-1 | 60 秒内支付次数达到 4
ABNORMAL_AMOUNT | key=O-6 | 金额 30000.0 超过阈值 10000.0
PAYMENT_TIMEOUT | key=O-2 | 创建后 5 分钟仍未支付

理解触发过程

O-1 在创建后 10 秒支付,算子会删除其超时定时器并清空待支付状态。O-2 没有支付事件,输入结束时水位线继续推进,5 分钟定时器触发并输出超时告警。

用户 U-1 在同一个 60 秒固定窗口内出现 4 次支付,因此只在计数首次达到阈值时告警。O-6 的金额超过阈值,直接输出金额告警,不必等待窗口或定时器。

生产边界与验证

事件时间定时器能否准时触发取决于水位线。Kafka 某个分区长期空闲时,应为水位线策略配置空闲检测,否则最小分区水位线可能拖住整个算子的定时器。

金额字段不应在生产系统中使用 double。可改用最小货币单位的 long,或使用定点小数类型,避免浮点误差影响阈值判断。固定窗口也存在边界效应;需要严格的任意 60 秒频率检测时,可改用滑动窗口、带时间戳的列表状态或 CEP。

告警应包含稳定的 alertId,例如由规则类型、业务键和窗口时间计算而来。下游数据库、Kafka 消费者或通知服务据此做幂等处理,避免作业恢复后重复通知。

按自己的节奏,确认真正理解。

目录与阅读设置

阅读设置

17

记录一个想法

检测交易异常

0 / 10000 · 笔记保存到账号。未提交的修改会在关闭、离开或刷新后丢失。

登录哪都通