[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$fIwnboToKh1ng32enCC-6mQ-qVuuIdSTYnBTBUctHA8Q":3},{"item":4,"related":43},{"id":5,"type":6,"title":7,"slug":8,"summary":9,"body":10,"coverUrl":11,"productScreenshots":12,"productLinks":13,"authorName":14,"authorUrl":15,"authorSubject":16,"category":17,"tags":22,"sourceLabel":35,"sourceName":35,"sourceUrl":35,"status":36,"seoTitle":35,"seoDescription":35,"canonicalUrl":35,"isFeatured":37,"sno":38,"sortOrder":39,"publishedAt":40,"updatedAt":41,"createdAt":42},"bce786d4-4fed-4186-a2b0-fee1a9762f28","article","大模型评测（LLM Evals）：为什么你的 AI 应用上线前必须做这件事","llm-evals-before-launch","模型「看起来能聊」和上生产是两回事。本文讲清 LLM Evals 为什么是 AI 应用的必需品：用带标准答案的题自动打分、建回归基线、用 LLM 当裁判，并给出 pytest 最小可运行示例与常见陷阱。","如果你做过 AI 应用，一定有过这种错觉：本地试聊几句，模型回答得头头是道，感觉「成了」。可一上线，用户随便问个边界问题，它就开始胡说、格式崩坏、甚至把上周还好好的功能改坏了。\n\n问题不在模型，在于你从来没用「可量化的标准」测过它。大模型评测（LLM Evals）就是解决这件事的：用一组带标准答案的题，自动跑、自动打分，把「行不行」变成数字。\n\n## 为什么需要 Evals\n\n靠人肉试聊有三个致命短板。第一是**回归陷阱**：你优化了一个 prompt，自己手感更好了，但可能悄悄搞砸了之前能答对的三类问题——没有对照基线，你根本发现不了。第二是**规模**：你不可能把上千种用户问法都手动试一遍。第三是**幻觉难察觉**：答案看起来通顺，事实却是错的，人眼抽查很容易漏。Evals 把「主观感觉」换成「可回归的指标体系」，每次改动都能看到分数涨跌。\n\n## Evals 的基本结构\n\n一套最小可用评测由四步串起来：准备数据集（输入 + 参考标准）、用被测模型跑出回答、用评分器打分、最后聚合出指标。评分器本身可以是硬规则、可以是另一个模型当裁判，也可以人工抽检。\n\n```mermaid\nflowchart LR\n    A[数据集 输入+参考答案] --> B[被测模型生成回答]\n    B --> C{评分器打分}\n    C -->|规则\u002F模型裁判\u002F人工| D[聚合指标 准确率\u002FF1\u002F通过率]\n    D --> E[对比基线 是否回归]\n```\n\n## 一个最小可运行的例子\n\n最朴素也最稳的做法，是用单元测试的框架（如 pytest）把「期望」写死：\n\n```python\nimport pytest\n\ncases = [\n    {\"q\": \"中国的首都是哪？\", \"expect\": \"北京\"},\n    {\"q\": \"1+1 等于几？\", \"expect\": \"2\"},\n]\n\ndef call_model(q: str) -> str:\n    # 这里换成你真实的模型调用\n    return \"北京\" if \"首都\" in q else \"2\"\n\n@pytest.mark.parametrize(\"c\", cases)\ndef test_basic(c):\n    got = call_model(c[\"q\"])\n    assert c[\"expect\"] in got, f\"期望含 {c['expect']}，实际 {got}\"\n```\n\n当你的场景变复杂（开放问答、长文本），再用「模型当裁判」（LLM-as-Judge）给定评分标准来打分，把分数也接进这套 pytest，就能在 CI 里跑回归。\n\n## 取舍与边界\n\n- **LLM 裁判有偏见**：它会偏爱长答案、会被措辞带偏，且每次调用要花钱、有延迟。关键场景一定要留人工抽检兜底。\n- **小样本不代表全量**：十道题全过，不等于线上万级流量没问题；数据集要持续收集真实 bad case 扩充。\n- **先建基线再优化**：没基线前别乱调 prompt，否则你永远不知道改动是变好还是变坏。\n- **指标要分层**：整体通过率之外，最好拆出「格式正确率」「事实准确率」「拒答恰当率」，定位问题更快。\n\n## Tips\n- 把最容易出错的 20 个真实问题整理成数据集，接进 pytest 跑通。\n- 把评测接进 CI：每次改 prompt \u002F 换模型，分数掉就拦下。\n- 开放问答类问题，引入 LLM-as-Judge，但保留 5% 人工抽检。\n- 线上一旦出现 bad case，立刻收录进数据集，让评测集跟着业务长。\n- 别追求「一个总分」，按格式 \u002F 事实 \u002F 安全分维度看，问题才好修。","https:\u002F\u002Foxqtewbrpuiouqqjrvdv.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fpublic-media\u002F2026-07-20\u002Fed78d51f-9c4c-4f9a-a9b7-fccde0c78f79.jpg",[],[],"Foundit AI","https:\u002F\u002Ffoundit.cn","f39339b1-aaa6-4e86-b0c2-a6e6a21113b5",{"id":18,"name":19,"slug":20,"description":21},"6179d3b6-dc34-4483-9ded-3cd9f1b37a47","科普","abbreviation","介绍各领域新兴概念",[23,27,31],{"id":24,"name":25,"slug":26},"7c76bfc2-f80f-4ee0-a95d-27bd8708b434","技术","slug",{"id":28,"name":29,"slug":30},"4c2bbea6-eab7-40a8-8447-1de478ff7749","分析","analyse",{"id":32,"name":33,"slug":34},"a202d639-99a6-488a-a712-4d4c6ffd7e15","开发","dev",null,"published",false,75,0,"2026-07-17T00:00:00.000Z","2026-07-20T01:19:14.428Z","2026-07-20T01:11:25.070Z",[44,54,63],{"id":45,"type":6,"title":46,"slug":47,"summary":48,"coverUrl":49,"authorName":50,"sno":51,"publishedAt":52,"createdAt":53},"9ccde95c-d754-4808-91f7-488f392e3eeb","你的品牌在AI眼里到底存不存在？这套系统说了算","automated-geo-monitoring-system","靠手动抽查来验证GEO效果，本质上是在跟概率玩游戏。赢一次，不代表能一直赢。","\u002Fuploads\u002F2026-08-07\u002Fdf111c0d-f14a-4b2a-8347-141e96b71654.jpg","Foundit",1,"2026-08-07T00:00:00.000Z","2026-08-07T04:31:30.843Z",{"id":55,"type":6,"title":56,"slug":57,"summary":58,"coverUrl":59,"authorName":14,"sno":60,"publishedAt":61,"createdAt":62},"544fc658-c911-4de6-93b0-d2520087119a","MCP：AI 的「USB-C」时刻","mcp-ai-usb-c-moment","以前每个 AI 应用都要为 GitHub、数据库、日历各写一套私有连接器，这是 M×N 的集成噩梦，直到 MCP 的出现","https:\u002F\u002Foxqtewbrpuiouqqjrvdv.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fpublic-media\u002F2026-07-19\u002Fc3c06d99-a0ac-40ac-a283-7e77aabb4c4c.jpg",46,"2026-07-20T00:00:00.000Z","2026-07-19T16:13:40.316Z",{"id":64,"type":6,"title":65,"slug":66,"summary":67,"coverUrl":68,"authorName":50,"sno":69,"publishedAt":70,"createdAt":71},"1bb735e6-efec-4ad5-9065-8ccfd38f5b5d","一张 JPEG 为什么会长出方块、蚊子噪点和白色光环？","why-jpeg-images-have-compression-artifacts","JPEG 通过色度抽样、分块变换、量化和熵编码大幅缩小照片，其中量化会永久舍弃部分细节。压缩过强或反复保存后，方块、边缘光环与蚊子噪点便会显现。本文从人眼偏好出发解释这些痕迹如何产生。","\u002Fuploads\u002F2026-08-14\u002F8011c7fa-472c-4599-a65a-c993b095375f.jpg",60,"2026-08-14T00:00:00.000Z","2026-08-14T03:06:16.293Z"]