[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$fpKf3UlrSVLFloUBezS6K8qXByswusD8rWsIJfUwl8pU":3},{"item":4,"related":44},{"id":5,"type":6,"title":7,"slug":8,"summary":9,"body":10,"coverUrl":11,"productScreenshots":12,"productLinks":13,"authorName":14,"authorUrl":15,"authorSubject":16,"category":17,"tags":22,"sourceLabel":35,"sourceName":36,"sourceUrl":37,"status":38,"seoTitle":7,"seoDescription":9,"canonicalUrl":35,"isFeatured":39,"sno":40,"sortOrder":41,"publishedAt":42,"updatedAt":43,"createdAt":43},"0d29aae9-204d-474e-8eaf-e7ba96fc8064","article","FlashAttention：注意力计算没变，为什么显卡能少搬很多数据？","flashattention-io-aware-transformer-attention","FlashAttention 通过分块计算与在线 softmax 减少 GPU 显存往返，并避免物化完整注意力矩阵。本文解释它为何加速长序列训练，同时指出密集注意力的二次计算量并未消失。","Transformer 的自注意力会比较序列中各个位置之间的关系。序列长度为 N 时，朴素实现往往要形成一个 N×N 的注意力矩阵，再计算 softmax 与加权求和。序列越长，矩阵越大；更隐蔽的瓶颈是，大量中间结果需要在 GPU 的高速片上存储与容量更大的显存之间来回搬运。\n\nFlashAttention 的关键不是改变注意力的数学答案，而是重新安排计算顺序。GPU 上的片上 SRAM 很快，但容量有限；高带宽显存（HBM）容量大一些，访问成本也更高。FlashAttention 把查询、键和值切分成小块，逐块计算注意力，并在线更新 softmax 所需的统计量。这样可以避免把完整的 N×N 注意力矩阵写入显存再读回来。\n\n为什么这种分块还能算出正确结果？softmax 的分母是所有键对应分数的指数和，表面上必须等看完全部位置才知道。算法利用分块累积的最大值、归一化因子与加权输出，在处理下一块时调整已有统计量，最终得到与标准注意力等价的结果（数值计算中允许浮点舍入差异）。反向传播也会重新计算部分中间值，以减少保存大量激活所需的显存。\n\n这是一种典型的“少搬数据、必要时多算一点”的硬件算法设计。GPU 上访存往往比算术操作更昂贵，因此多做一些局部计算，反而可以缩短总体耗时并减少显存占用。尤其在长序列和训练场景中，避免存储完整注意力矩阵，能让更大的批量或更长上下文成为可能。\n\n不过 FlashAttention 并没有消除密集注意力的二次计算量。每个查询位置仍要与很多键位置计算关联，序列长度翻倍时，主要计算量仍大约按平方增长。它主要优化的是 I\u002FO 复杂度和内存占用，不是把所有长上下文问题变成线性复杂度；实际加速也依赖 GPU 架构、数据类型、序列长度及实现版本。\n\n可以把它想成不用把整本书复印到桌面再算，而是每次取几页，在桌面上更新总账，然后把过期的草稿丢掉。桌面空间小但快，仓库（显存）大却不适合频繁往返。FlashAttention 的贡献，是让注意力算法顺着 GPU 的内存层次来安排工作，而不是把硬件当作无限大的平面黑板。\n\n来源：[Dao 等：FlashAttention，NeurIPS 2022](https:\u002F\u002Farxiv.org\u002Fabs\u002F2205.14135)","\u002Fuploads\u002F2026-10-13\u002Faf2e6109-e447-48bf-9729-874072603e3a.jpg",[],[],"Foundit","https:\u002F\u002Ffoundit.cn","foundit-ai-editorial",{"id":18,"name":19,"slug":20,"description":21},"6179d3b6-dc34-4483-9ded-3cd9f1b37a47","科普","abbreviation","介绍各领域新兴概念",[23,27,31],{"id":24,"name":25,"slug":26},"7c76bfc2-f80f-4ee0-a95d-27bd8708b434","技术","slug",{"id":28,"name":29,"slug":30},"88d2bc27-0e0f-468a-b907-2991cb97b87b","人工智能","ai",{"id":32,"name":33,"slug":34},"4c2bbea6-eab7-40a8-8447-1de478ff7749","分析","analyse",null,"NeurIPS 2022 \u002F arXiv","https:\u002F\u002Farxiv.org\u002Fabs\u002F2205.14135","published",false,50,0,"2026-10-13T00:00:00.000Z","2026-10-13T02:14:02.226Z",[45,54,62],{"id":46,"type":6,"title":47,"slug":48,"summary":49,"coverUrl":50,"authorName":14,"sno":51,"publishedAt":52,"createdAt":53},"872963cd-242c-4912-a3dc-9497386d4772","Certificate Transparency：TLS 证书为什么要公开留日志？","certificate-transparency-public-audit-logs","证书透明度通过公开、可验证的日志提升 TLS 证书签发的可观察性。本文说明监控者如何发现意外证书，也厘清 CT 不会自动阻止误发。","\u002Fuploads\u002F2026-10-06\u002Fa2b8f9d0-fb90-4f19-ae2b-22cdebd395f2.jpg",43,"2026-10-06T00:00:00.000Z","2026-10-06T07:51:39.208Z",{"id":55,"type":6,"title":56,"slug":57,"summary":58,"coverUrl":59,"authorName":14,"sno":60,"publishedAt":42,"createdAt":61},"3eee805c-a6c6-438b-b41c-bb67359211f6","HNSW：向量数据库怎样在“小世界图”里快速找相似项？","hnsw-approximate-nearest-neighbor-search","HNSW 把向量组织为多层可导航图，从稀疏高层逐步下探到局部候选区域，以较少距离计算近似查找邻居。本文讲解图搜索原理，以及召回率、延迟、构建和内存之间的权衡。","\u002Fuploads\u002F2026-10-13\u002Feb9f9a0c-fcbb-4323-86ea-58d5ca17c9f2.jpg",47,"2026-10-13T02:14:06.187Z",{"id":63,"type":6,"title":64,"slug":65,"summary":66,"coverUrl":67,"authorName":14,"sno":40,"publishedAt":68,"createdAt":69},"f00e5274-8e0b-40fe-af3b-b6a8d0183b9c","一张贴纸就能骗过视觉 AI？对抗样本不是魔法","adversarial-examples-fool-visual-ai","人眼仍能认出的物体，经过精心设计的微小扰动或标记后，机器却可能改变判断。这类输入称为对抗样本。本文解释攻击者如何利用模型的决策边界，现实攻击为何比实验更难，以及为什么目前不存在一劳永逸的防御。","\u002Fuploads\u002F2026-09-08\u002F151f887b-c5f1-4647-b369-cdda8a1e1b4f.jpg","2026-09-03T00:00:00.000Z","2026-08-14T03:06:09.765Z"]