[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$fOyrRuBPH5cI5GbEfYaj67S1h--gU4_ByDRl218K8YZc":3},{"item":4,"related":43},{"id":5,"type":6,"title":7,"slug":8,"summary":9,"body":10,"coverUrl":11,"productScreenshots":12,"productLinks":13,"authorName":14,"authorUrl":15,"authorSubject":16,"category":17,"tags":22,"sourceLabel":35,"sourceName":36,"sourceUrl":36,"status":37,"seoTitle":36,"seoDescription":36,"canonicalUrl":36,"isFeatured":38,"sno":39,"sortOrder":40,"publishedAt":41,"updatedAt":41,"createdAt":42},"f8b9ea19-4820-4ab7-804a-918726bfb0dd","article","模型蒸馏：让小模型「偷师」大模型，把强者经验压进手机","model-distillation-teacher-student","大模型贵、小模型笨，蒸馏让小模型学走大模型的「隐藏知识」。本文用师徒制讲清软标签与温度的作用，以及 QLoRA+蒸馏如何把几百亿参数压到手机本地跑的取舍。","大模型聪明但贵，小模型便宜却常犯傻。有没有办法让小模型「偷师」大模型？这就是模型蒸馏（Distillation）在干的事。\n\n经典做法像师徒制。先用大模型（教师）对训练数据产出「软标签」——不是简单的「这是猫 \u002F 不是猫」，而是「猫 0.7、狗 0.2、狐狸 0.1」这种带温度的概率分布。这些软标签藏着教师模型学到的「类与类之间的微妙关系」：猫和狗比猫和汽车更近。小模型（学生）在学习时，不只拟合正确答案，还去贴近教师的软标签，于是把那些「隐藏知识」一并学走。\n\n```mermaid\nflowchart LR\n    T[教师模型] --> S[软标签 概率分布]\n    S --> St[学生模型]\n    D[真实标签] --> St\n```\n\n训练目标通常是两者的加权：\n\n```python\nloss = alpha * KL(学生软标签, 教师软标签) + (1 - alpha) * CE(学生输出, 真实标签)\n```\n\n训练时有个关键旋钮叫「温度（temperature）」：调高温度，软标签更平滑，类间关系更明显，学生更容易学到；预测时再把温度调回 1。\n\n现实里蒸馏为什么香？比如把几百亿参数的模型压到几亿，塞进手机本地跑，隐私不出设备、还免了每次调用的服务器账单。QLoRA + 蒸馏的组合，已经能让一张普通显卡「炼」出可用的小模型；更有「无数据蒸馏」，用教师自己生成训练样本，连原始数据都不需要。\n\n当然有代价：学生上限受教师天花板限制，且教师本身得够强、够稳。\n\n实操上，温度常取 2~4 来生成软标签，学生用同样的温度去匹配，推理时再归 1；教师越强、与学生差距越大，蒸馏收益越明显，但教师的错误也会被一并「传染」下来。典型的 DistilBERT 就是用蒸馏把 BERT 压到约 40% 的体积、保留近 97% 的效果，成了不少生产环境的默认选择。\n\n蒸馏不是点金术，是「把强者的经验压缩给弱者」的实在工程。","https:\u002F\u002Foxqtewbrpuiouqqjrvdv.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fpublic-media\u002F2026-07-21\u002Fe620dfc1-1377-486e-876d-12efe02da22e.jpg",[],[],"Foundit AI","https:\u002F\u002Ffoundit.cn","f39339b1-aaa6-4e86-b0c2-a6e6a21113b5",{"id":18,"name":19,"slug":20,"description":21},"6179d3b6-dc34-4483-9ded-3cd9f1b37a47","科普","abbreviation","介绍各领域新兴概念",[23,27,31],{"id":24,"name":25,"slug":26},"88d2bc27-0e0f-468a-b907-2991cb97b87b","人工智能","ai",{"id":28,"name":29,"slug":30},"7c76bfc2-f80f-4ee0-a95d-27bd8708b434","技术","slug",{"id":32,"name":33,"slug":34},"4c2bbea6-eab7-40a8-8447-1de478ff7749","分析","analyse","资料来源",null,"published",false,70,0,"2026-07-21T06:35:41.144Z","2026-07-21T06:25:01.094Z",[44,54,63],{"id":45,"type":6,"title":46,"slug":47,"summary":48,"coverUrl":49,"authorName":50,"sno":51,"publishedAt":52,"createdAt":53},"f00e5274-8e0b-40fe-af3b-b6a8d0183b9c","一张贴纸就能骗过视觉 AI？对抗样本不是魔法","adversarial-examples-fool-visual-ai","人眼仍能认出的物体，经过精心设计的微小扰动或标记后，机器却可能改变判断。这类输入称为对抗样本。本文解释攻击者如何利用模型的决策边界，现实攻击为何比实验更难，以及为什么目前不存在一劳永逸的防御。","\u002Fuploads\u002F2026-09-08\u002F151f887b-c5f1-4647-b369-cdda8a1e1b4f.jpg","Foundit",50,"2026-09-03T00:00:00.000Z","2026-08-14T03:06:09.765Z",{"id":55,"type":6,"title":56,"slug":57,"summary":58,"coverUrl":59,"authorName":50,"sno":60,"publishedAt":61,"createdAt":62},"305492d4-c146-456a-b341-31140ab9cafd","天气预报不再一格一格算空气，AI 是怎么预测风暴的？","how-ai-weather-forecasting-works","传统数值预报依据物理方程推进大气状态，AI 天气模型则从历史观测与再分析数据中学习状态如何演变。本文以 GraphCast 为例，解释图神经网络如何快速预测全球天气、它与传统方法如何协作，以及极端天气仍有哪些难点。","\u002Fuploads\u002F2026-08-16\u002Fa8b5ddad-d6db-4d12-a159-87a6c5309082.jpg",60,"2026-08-16T00:00:00.000Z","2026-08-14T03:06:12.102Z",{"id":64,"type":6,"title":65,"slug":66,"summary":67,"coverUrl":68,"authorName":50,"sno":60,"publishedAt":69,"createdAt":70},"7f962194-e0c6-4072-9b50-c70054beb0e5","同一个问题问三遍，AI 为什么会给出三个答案？","why-ai-gives-different-answers-sampling","大模型每次回答都在从候选词中继续选择，而不是从数据库里取出一段固定文字。温度、Top-p 和随机采样共同决定回答更稳定还是更有变化。本文用抽签与岔路的比喻，解释 AI 的随机性从哪里来，以及什么时候应该追求一致。","\u002Fuploads\u002F2026-08-14\u002Fb322d338-b5c0-48c7-addf-fbd91421e316.jpg","2026-08-14T00:00:00.000Z","2026-08-14T03:06:07.351Z"]