大数据与流处理 / 系统课时
检测交易异常
检测交易异常
本课把支付超时、短时高频和异常金额三类规则落成一个可运行的 Flink 作业。完成后,你能够为不同规则选择正确的分区键,使用键控状态保存上下文,并用事件时间定时器在条件满足时输出统一告警。
学习目标
- 为支付超时和用户行为规则选择不同的
keyBy键 - 使用
ValueState与事件时间定时器保存并清理检测上下文 - 验证高频、超额和超时告警是否按预期产生
规则拆分与键选择
三类异常不能强行放进同一个键控算子。支付超时需要按 orderId 聚合创建与支付事件;短时高频需要按 userId 统计;异常金额虽然不依赖历史状态,但可与用户行为检测放在同一分支中输出。
| 规则 | 分区键 | 状态 | 定时器 | 触发条件 |
|---|---|---|---|---|
| 支付超时 | orderId |
待支付事件、超时时刻 | 事件时间 | 创建后 5 分钟仍未支付 |
| 短时高频 | userId |
窗口计数、窗口结束时间 | 事件时间 | 60 秒内支付达到 4 次 |
| 异常金额 | userId |
无必需状态 | 无 | 单笔支付金额超过 10000 |

实现状态与定时器检测
下面的完整示例使用固定测试事件。项目需引入与集群版本一致的 Flink DataStream 依赖;示例时间戳是确定值,因此每次运行都能复现相同的三类告警。
java
package com.anystack.flink.trade;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.OpenContext;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
import org.apache.flink.util.Collector;
import java.io.Serializable;
import java.time.Duration;
public class TradeAnomalyJob {
private static final long PAYMENT_TIMEOUT_MS = Duration.ofMinutes(5).toMillis();
private static final long FREQUENCY_WINDOW_MS = Duration.ofSeconds(60).toMillis();
private static final int FREQUENCY_THRESHOLD = 4;
private static final double AMOUNT_THRESHOLD = 10_000D;
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
long base = 1_700_000_000_000L;
DataStream<TradeEvent> events = env.fromElements(
new TradeEvent("O-1", "U-1", "CREATE", 100D, base),
new TradeEvent("O-2", "U-2", "CREATE", 200D, base + 5_000),
new TradeEvent("O-1", "U-1", "PAY", 100D, base + 10_000),
new TradeEvent("O-3", "U-1", "PAY", 10D, base + 20_000),
new TradeEvent("O-4", "U-1", "PAY", 20D, base + 30_000),
new TradeEvent("O-5", "U-1", "PAY", 40D, base + 40_000),
new TradeEvent("O-6", "U-3", "PAY", 30_000D, base + 50_000)
).assignTimestampsAndWatermarks(
WatermarkStrategy.<TradeEvent>forBoundedOutOfOrderness(
Duration.ofSeconds(2))
.withTimestampAssigner((event, previous) -> event.eventTime)
);
DataStream<Alert> timeoutAlerts = events
.keyBy(event -> event.orderId)
.process(new PaymentTimeoutDetector());
DataStream<Alert> behaviorAlerts = events
.keyBy(event -> event.userId)
.process(new BehaviorDetector());
timeoutAlerts
.union(behaviorAlerts)
.map(Alert::toString)
.print();
env.execute("trade-anomaly-detection");
}
public static class PaymentTimeoutDetector
extends KeyedProcessFunction<String, TradeEvent, Alert> {
private transient ValueState<TradeEvent> pendingCreate;
private transient ValueState<Long> timeoutAt;
@Override
public void open(OpenContext openContext) {
pendingCreate = getRuntimeContext().getState(
new ValueStateDescriptor<>("pending-create", TradeEvent.class));
timeoutAt = getRuntimeContext().getState(
new ValueStateDescriptor<>("timeout-at", Long.class));
}
@Override
public void processElement(
TradeEvent event,
Context ctx,
Collector<Alert> out) throws Exception {
if ("CREATE".equals(event.eventType)) {
Long oldTimer = timeoutAt.value();
if (oldTimer != null) {
ctx.timerService().deleteEventTimeTimer(oldTimer);
}
long timer = event.eventTime + PAYMENT_TIMEOUT_MS;
pendingCreate.update(event);
timeoutAt.update(timer);
ctx.timerService().registerEventTimeTimer(timer);
return;
}
if ("PAY".equals(event.eventType) && pendingCreate.value() != null) {
Long timer = timeoutAt.value();
if (timer != null) {
ctx.timerService().deleteEventTimeTimer(timer);
}
pendingCreate.clear();
timeoutAt.clear();
}
}
@Override
public void onTimer(
long timestamp,
OnTimerContext ctx,
Collector<Alert> out) throws Exception {
TradeEvent create = pendingCreate.value();
Long expectedTimer = timeoutAt.value();
if (create != null && expectedTimer != null && timestamp == expectedTimer) {
out.collect(new Alert(
"PAYMENT_TIMEOUT",
create.orderId,
"创建后 5 分钟仍未支付"));
pendingCreate.clear();
timeoutAt.clear();
}
}
}
public static class BehaviorDetector
extends KeyedProcessFunction<String, TradeEvent, Alert> {
private transient ValueState<Integer> windowCount;
private transient ValueState<Long> windowEnd;
@Override
public void open(OpenContext openContext) {
windowCount = getRuntimeContext().getState(
new ValueStateDescriptor<>("window-count", Integer.class));
windowEnd = getRuntimeContext().getState(
new ValueStateDescriptor<>("window-end", Long.class));
}
@Override
public void processElement(
TradeEvent event,
Context ctx,
Collector<Alert> out) throws Exception {
if (!"PAY".equals(event.eventType)) {
return;
}
if (event.amount > AMOUNT_THRESHOLD) {
out.collect(new Alert(
"ABNORMAL_AMOUNT",
event.orderId,
"金额 " + event.amount + " 超过阈值 " + AMOUNT_THRESHOLD));
}
Long end = windowEnd.value();
if (end == null || event.eventTime >= end) {
if (end != null) {
ctx.timerService().deleteEventTimeTimer(end);
}
long newEnd = event.eventTime + FREQUENCY_WINDOW_MS;
windowCount.update(1);
windowEnd.update(newEnd);
ctx.timerService().registerEventTimeTimer(newEnd);
return;
}
Integer current = windowCount.value();
int next = (current == null ? 0 : current) + 1;
windowCount.update(next);
if (next == FREQUENCY_THRESHOLD) {
out.collect(new Alert(
"HIGH_FREQUENCY",
event.userId,
"60 秒内支付次数达到 " + next));
}
}
@Override
public void onTimer(
long timestamp,
OnTimerContext ctx,
Collector<Alert> out) throws Exception {
Long end = windowEnd.value();
if (end != null && timestamp == end) {
windowCount.clear();
windowEnd.clear();
}
}
}
public static class TradeEvent implements Serializable {
public String orderId;
public String userId;
public String eventType;
public double amount;
public long eventTime;
public TradeEvent() {
}
public TradeEvent(
String orderId,
String userId,
String eventType,
double amount,
long eventTime) {
this.orderId = orderId;
this.userId = userId;
this.eventType = eventType;
this.amount = amount;
this.eventTime = eventTime;
}
}
public static class Alert implements Serializable {
public String type;
public String key;
public String message;
public Alert() {
}
public Alert(String type, String key, String message) {
this.type = type;
this.key = key;
this.message = message;
}
@Override
public String toString() {
return type + " | key=" + key + " | " + message;
}
}
}预期输出如下,两个分支在 union 后的行顺序可能不同。
text
HIGH_FREQUENCY | key=U-1 | 60 秒内支付次数达到 4
ABNORMAL_AMOUNT | key=O-6 | 金额 30000.0 超过阈值 10000.0
PAYMENT_TIMEOUT | key=O-2 | 创建后 5 分钟仍未支付理解触发过程
O-1 在创建后 10 秒支付,算子会删除其超时定时器并清空待支付状态。O-2 没有支付事件,输入结束时水位线继续推进,5 分钟定时器触发并输出超时告警。
用户 U-1 在同一个 60 秒固定窗口内出现 4 次支付,因此只在计数首次达到阈值时告警。O-6 的金额超过阈值,直接输出金额告警,不必等待窗口或定时器。
生产边界与验证
事件时间定时器能否准时触发取决于水位线。Kafka 某个分区长期空闲时,应为水位线策略配置空闲检测,否则最小分区水位线可能拖住整个算子的定时器。
金额字段不应在生产系统中使用 double。可改用最小货币单位的 long,或使用定点小数类型,避免浮点误差影响阈值判断。固定窗口也存在边界效应;需要严格的任意 60 秒频率检测时,可改用滑动窗口、带时间戳的列表状态或 CEP。
告警应包含稳定的 alertId,例如由规则类型、业务键和窗口时间计算而来。下游数据库、Kafka 消费者或通知服务据此做幂等处理,避免作业恢复后重复通知。
按自己的节奏,确认真正理解。