机器学习与深度学习 / 系统课时

封装推荐 API 并压测延迟

封装推荐 API 并压测延迟

本课把离线推荐链路封装成可压测的在线 API,并加入缓存降级与超时保护。完成后你能用并发请求测出吞吐、P95 延迟和降级比例,并按候选检索、特征读取、模型推理三个阶段定位主要耗时。

学习目标

  • 封装一个包含召回、特征、排序和降级路径的推荐接口。
  • 用并发压测统计吞吐、P95 延迟和错误率。
  • 根据分阶段耗时判断瓶颈,并验证超时与缓存降级是否真正生效。

先把在线链路拆成可观测阶段

推荐接口的总延迟不能只看一个总数。至少要把请求拆成候选检索、特征读取、模型推理和组装返回四段,并把每段耗时写进响应头或监控指标。这样 P95 升高时,才能知道是检索慢、特征存储抖动,还是模型推理拖住了请求。

在线推荐请求的数据流与降级路径

下面的示例用 FastAPI 模拟三个阶段,并把 80 ms 设为总超时预算。若主链路超时,就返回预置缓存结果,同时通过 X-Degraded 标识降级。

python
import asyncio
import time
from fastapi import FastAPI, Response

app = FastAPI()
CACHE = {"u1": ["i-cache-1", "i-cache-2", "i-cache-3"]}

async def recall_candidates(user_id: str) -> list[str]:
    await asyncio.sleep(0.010)
    return [f"i-{n}" for n in range(1, 51)]

async def load_features(user_id: str, items: list[str]) -> dict[str, float]:
    await asyncio.sleep(0.020)
    return {item: (hash(user_id + item) % 1000) / 1000 for item in items}

async def rank_items(features: dict[str, float]) -> list[str]:
    await asyncio.sleep(0.030)
    return [item for item, _ in sorted(features.items(), key=lambda x: x[1], reverse=True)[:10]]

async def recommend_main(user_id: str) -> tuple[list[str], dict[str, float]]:
    timings: dict[str, float] = {}

    started = time.perf_counter()
    candidates = await recall_candidates(user_id)
    timings["candidate_ms"] = (time.perf_counter() - started) * 1000

    started = time.perf_counter()
    features = await load_features(user_id, candidates)
    timings["feature_ms"] = (time.perf_counter() - started) * 1000

    started = time.perf_counter()
    items = await rank_items(features)
    timings["model_ms"] = (time.perf_counter() - started) * 1000
    return items, timings

@app.get("/recommend/{user_id}")
async def recommend(user_id: str, response: Response):
    total_started = time.perf_counter()
    degraded = False

    try:
        items, timings = await asyncio.wait_for(recommend_main(user_id), timeout=0.080)
    except asyncio.TimeoutError:
        items = CACHE.get(user_id, ["i-hot-1", "i-hot-2", "i-hot-3"])
        timings = {"candidate_ms": -1, "feature_ms": -1, "model_ms": -1}
        degraded = True

    total_ms = (time.perf_counter() - total_started) * 1000
    response.headers["X-Degraded"] = str(degraded).lower()
    response.headers["X-Total-Ms"] = f"{total_ms:.2f}"
    return {"userId": user_id, "items": items, "timings": timings}

生效行为:正常情况下返回模型排序结果;总耗时超过 80 ms 时,不继续等待主链路,而是返回缓存或热门结果。实际生产中应把总预算进一步拆成各阶段超时预算,避免前一阶段耗尽后续时间。

启动并验证接口

安装依赖并启动服务:

bash
python -m pip install fastapi uvicorn
uvicorn app:app --host 127.0.0.1 --port 8000

预期结果:

text
INFO:     Uvicorn running on http://127.0.0.1:8000

端口与日志时间等动态值可能不同。另开终端请求接口:

bash
curl -i http://127.0.0.1:8000/recommend/u1

预期结果:

text
HTTP/1.1 200 OK
x-degraded: false
x-total-ms: <TIME>

{"userId":"u1","items":["<ID>"],"timings":{"candidate_ms":<TIME>,"feature_ms":<TIME>,"model_ms":<TIME>}}

条目 ID、阶段耗时和总耗时会随运行环境变化,但正常请求应返回 200,并且 x-degraded 为 false。

用并发请求测吞吐和 P95

压测脚本不需要复杂工具也能得到第一版基线。下面用 100 个并发槽发出 1000 个请求,统计吞吐、P95、错误数和降级数。

python
import asyncio
import time
import httpx

URL = "http://127.0.0.1:8000/recommend/u1"
TOTAL = 1000
CONCURRENCY = 100

async def one_request(client: httpx.AsyncClient, sem: asyncio.Semaphore):
    async with sem:
        started = time.perf_counter()
        try:
            response = await client.get(URL, timeout=2.0)
            latency_ms = (time.perf_counter() - started) * 1000
            return latency_ms, response.status_code, response.headers.get("x-degraded") == "true"
        except httpx.HTTPError:
            latency_ms = (time.perf_counter() - started) * 1000
            return latency_ms, 0, False

async def main():
    sem = asyncio.Semaphore(CONCURRENCY)
    started = time.perf_counter()

    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*(one_request(client, sem) for _ in range(TOTAL)))

    elapsed = time.perf_counter() - started
    latencies = sorted(item[0] for item in results)
    p95 = latencies[max(0, int(len(latencies) * 0.95) - 1)]
    errors = sum(1 for _, status, _ in results if status != 200)
    degraded = sum(1 for _, _, is_degraded in results if is_degraded)

    print(f"requests={TOTAL}")
    print(f"throughput_rps={TOTAL / elapsed:.1f}")
    print(f"p95_ms={p95:.1f}")
    print(f"errors={errors}")
    print(f"degraded={degraded}")

asyncio.run(main())

先安装依赖并运行:

bash
python -m pip install httpx
python load_test.py

预期结果:

text
requests=1000
throughput_rps=<RPS>
p95_ms=<TIME>
errors=0
degraded=<COUNT>

<RPS>、<TIME> 和 <COUNT> 受 CPU、Python 调度和机器负载影响,不能写死。压测时应同时观察服务端阶段耗时,否则只能知道“慢”,不能知道“慢在哪里”。

按阶段耗时定位瓶颈

若 candidate_ms 的 P95 明显高于其他阶段,优先检查向量库或倒排检索的索引命中、连接池和查询上限。若 feature_ms 抖动,重点看特征缓存命中率、批量读取方式和远程存储 RT。若 model_ms 占比最高,则检查批处理、模型大小、线程数以及是否发生 CPU 或 GPU 排队。

一个实用判断是把总预算倒推到各阶段。例如接口目标 P95 不超过 100 ms,可先给召回 25 ms、特征 25 ms、模型 35 ms、网络与组装 15 ms。任何阶段长期吃满预算,都应独立优化,而不是只扩大总超时。

验证超时保护和缓存降级

验证降级不能靠“代码里写了 except”就算完成。把 rank_items 的休眠临时改成 100 ms,再请求接口;此时主链路必然超过 80 ms,响应头应切换为 x-degraded: true,并返回缓存结果。

修改后再次执行:

bash
curl -i http://127.0.0.1:8000/recommend/u1

预期结果:

text
HTTP/1.1 200 OK
x-degraded: true
x-total-ms: <TIME>

{"userId":"u1","items":["i-cache-1","i-cache-2","i-cache-3"],"timings":{"candidate_ms":-1,"feature_ms":-1,"model_ms":-1}}

如果高并发下错误率上升但降级数始终为 0,说明请求可能在进入超时保护前就被连接池、线程池或下游队列拖死。此时要把并发上限、下游超时和熔断边界一起检查,确保降级路径真的位于故障传播之前。

按自己的节奏,确认真正理解。

目录与阅读设置

阅读设置

17

记录一个想法

封装推荐 API 并压测延迟

0 / 10000 · 笔记保存到账号。未提交的修改会在关闭、离开或刷新后丢失。

登录哪都通