AI Agent / 系统课时
处理超时与重复调用
处理超时与重复调用
工具调用最危险的情况不是“明确失败”,而是请求已经产生副作用、响应却因超时丢失。正确做法是把超时、有限重试、幂等键和熔断组合起来,让重试可控、重复调用不重复执行副作用。
学习目标
- 判断超时后哪些调用可以重试,哪些调用必须先具备幂等能力。
- 用同一个幂等键抵御“已成功但响应丢失”导致的重复执行。
- 用有限重试和熔断阻止故障期间请求持续放大。
先识别超时的危险点
超时只说明调用方没有在期限内拿到结果,并不能证明工具端没有执行。天气、库存等只读请求可在故障可恢复时进行有限重试,并受调用预算和限流约束。扣款、下单、发消息等有副作用的操作,如果每次重试都生成新的请求身份,就可能执行两次。
为一次逻辑动作生成一个业务幂等键,并在传输重试、进程重启和状态核对时复用。模型调用 ID 用于协议配对,重生成时可能变化,不能代替业务幂等键。服务端需要将键作用域、请求参数和执行结果关联,并在同一原子提交中处理本地副作用与记录。

用一个主示例串起四道保护
把代码保存为 credit_retries.py,与上一课的 credit_policy.py 放在同一目录。SQLite 示例使用临时内存库。事务内校验权限、审批和余额,并写入收支及幂等记录。响应丢失发生在提交之后,下一次调用应读回原记录。示例用串行熔断器演示连续超时、拒绝新请求和冷却后探测。
python
import sqlite3
import time
from credit_policy import Approval, CreditError, TransferContext, validate_transfer
SCHEMA = """
CREATE TABLE accounts (
user_id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL,
balance INTEGER NOT NULL CHECK(typeof(balance) = 'integer' AND balance >= 0)
);
CREATE TABLE transfers (
tenant_id TEXT NOT NULL, actor_id TEXT NOT NULL, operation_key TEXT NOT NULL,
target_user_id TEXT NOT NULL, amount INTEGER NOT NULL,
PRIMARY KEY (tenant_id, actor_id, operation_key)
);
"""
def transfer_once(db, context, approval, args, now):
if db.in_transaction:
raise RuntimeError("transfer needs an idle connection")
with db:
db.execute("BEGIN IMMEDIATE")
tenants = dict(db.execute("SELECT user_id, tenant_id FROM accounts"))
target, amount = validate_transfer(args, context, approval, tenants, now)
key = (context.tenant_id, context.actor_id, context.operation_key)
previous = db.execute(
"""SELECT target_user_id, amount FROM transfers
WHERE tenant_id = ? AND actor_id = ? AND operation_key = ?""",
key,
).fetchone()
if previous is not None:
if previous != (target, amount):
raise CreditError("IDEMPOTENCY_CONFLICT")
return {"operation_key": context.operation_key, "transferred": amount}
debit = db.execute(
"UPDATE accounts SET balance = balance - ? WHERE user_id = ? AND balance >= ?",
(amount, context.actor_id, amount),
)
if debit.rowcount != 1:
raise CreditError("INSUFFICIENT_CREDITS")
db.execute("UPDATE accounts SET balance = balance + ? WHERE user_id = ?", (amount, target))
db.execute("INSERT INTO transfers VALUES (?, ?, ?, ?, ?)", (*key, target, amount))
return {"operation_key": context.operation_key, "transferred": amount}
class CircuitOpenError(RuntimeError):
pass
class Circuit:
def __init__(self, threshold=2, cooldown=5):
self.threshold = threshold
self.cooldown = cooldown
self.failures = 0
self.open_until = None
def call(self, operation, now):
if self.open_until is not None and now < self.open_until:
raise CircuitOpenError("circuit open")
try:
result = operation()
except TimeoutError:
self.failures += 1
if self.failures >= self.threshold:
self.open_until = now + self.cooldown
raise
self.failures = 0
self.open_until = None
return result
def retry(operation, circuit, clock=time.monotonic, pause=time.sleep):
try:
return circuit.call(operation, clock())
except TimeoutError:
pause(0.01)
return circuit.call(operation, clock())
if __name__ == "__main__":
db = sqlite3.connect(":memory:", isolation_level=None)
db.executescript(SCHEMA)
with db:
db.executemany("INSERT INTO accounts VALUES (?, ?, ?)", [
("alice", "team-a", 100), ("bob", "team-a", 0),
])
context = TransferContext("team-a", "alice", frozenset({"credit:transfer"}), "op-1")
approval = Approval("team-a", "alice", "op-1", "bob", 50, 200)
args = {"target_user_id": "bob", "amount": 50}
attempts = 0
def unreliable():
global attempts
attempts += 1
result = transfer_once(db, context, approval, args, now=100)
if attempts == 1:
raise TimeoutError("response lost after commit")
return result
print(retry(unreliable, Circuit()))
print(list(db.execute("SELECT user_id, balance FROM accounts ORDER BY user_id")))
print("ledger rows:", db.execute("SELECT count(*) FROM transfers").fetchone()[0])
db.close()
def unavailable():
raise TimeoutError("upstream timeout")
circuit = Circuit()
for now in (0, 1, 2, 6):
try:
operation = unavailable if now < 6 else lambda: "probe passed"
print(circuit.call(operation, now))
except TimeoutError:
print("timeout")
except CircuitOpenError:
print("circuit open")预期输出
text
{'operation_key': 'op-1', 'transferred': 50}
[('alice', 50), ('bob', 50)]
ledger rows: 1
timeout
timeout
circuit open
probe passed把课堂机制用到真实调用
- 单次超时和总截止时间分别控制。达到截止时间后停止派发新动作。超时后的写操作可能已经提交。若外部系统不承诺幂等,应先查询状态、核对凭证或转人工处理。
- 确认工具端支持同一键、同一参数的安全重放后,再对写调用退避重试。正式服务还应限制最大尝试次数,采用带抖动的退避,并按接口契约处理限流及
Retry-After。参数、权限或审批失败不计入上游瞬时故障。 - SQLite 示例要求独占使用一条空闲连接,由写锁保护幂等检查与副作用。权限和审批来自服务端测试上下文。真实服务使用持久库并在提交前按一致性要求读取最新权限和审批。若副作用发生在外部服务,本地事务无法覆盖远端提交,需要外部幂等接口或可核验的状态查询。审批过期后可走另行授权的状态查询,不能重新发起写入。
- 熔断按工具或上游故障域计数。串行示例中冷却后只有当前调用能探测。并发实现需要同步限制探测数量,并在探测成功后才关闭熔断。熔断不会撤销已发出的请求。
验证保护是否生效
至少测试提交后丢响应、并发使用同一键、同键不同参数、余额不足回滚、持续故障和冷却后探测。分别核对调用返回、账户余额及账本记录。一次结果文本相同并不能说明副作用只发生了一次。
按自己的节奏,确认真正理解。