[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$fCWI95AAz9WUfRvAJvjWWd7Dc2ertQR5nWIJnOltrBqg":3},{"item":4,"related":44},{"id":5,"type":6,"title":7,"slug":8,"summary":9,"body":10,"coverUrl":11,"productScreenshots":12,"productLinks":13,"authorName":14,"authorUrl":15,"authorSubject":16,"category":17,"tags":22,"sourceLabel":35,"sourceName":35,"sourceUrl":36,"status":37,"seoTitle":7,"seoDescription":9,"canonicalUrl":38,"isFeatured":39,"sno":40,"sortOrder":41,"publishedAt":42,"updatedAt":43,"createdAt":43},"b3021f89-78c0-4d93-bc30-cea222b30f37","article","ORCA：大模型服务怎样在每生成一个 token 后重排队伍？","orca-iteration-level-scheduling-llm-serving","ORCA 提出 iteration-level scheduling：大模型每轮生成后都重新安排批次，让刚完成的请求退出、新请求及时进入。本文用公交发车与流水线的比喻解释连续批处理的效率，也讨论调度器如何平衡吞吐与等待时间。","大模型服务常常同时处理许多用户请求。传统批处理可能把一组请求一起送入 GPU，但生成长度不同：有的回答很快结束，有的还要继续很久。若必须等整批最慢的请求完成才能换入新请求，GPU 就可能出现空档，排队的用户也要等。ORCA 提出的 iteration-level scheduling（迭代级调度）改变了重新组批的时机。\n\n## 每轮生成后重新看队伍\n\n自回归模型一次迭代通常为每个活跃请求生成一个或一小段 token。ORCA 的调度思路是在每次模型迭代之后重新评估批次：已经结束的请求可以退出，新请求有机会加入，持续生成的请求继续运行。批次不再必须从头到尾绑定同一组用户。\n\n可以把它想成公交车每到一站就重新分配座位：下车的人空出座位，站台上的新乘客可以补上，而长途乘客不需要等整辆车绕完一圈再发车。对在线推理系统而言，这种持续批处理能更灵活地利用 GPU，减少由于输出长度差异造成的空闲。\n\n## 吞吐提高，还要管理公平与延迟\n\n迭代级调度要求服务端把请求状态、KV Cache 和批次生命周期管理得更细。每轮加入的新请求可能需要预填充输入上下文，老请求则进入逐 token 解码；两类计算的资源需求不同。调度器必须处理显存容量、请求优先级、超时取消和批次切换开销。\n\n连续把新请求塞进 GPU，有机会提升总体吞吐，却不自动保证单个用户更快完成。若系统只追求利用率，长请求或低优先级请求可能等得更久；反之，过度保护单个请求又会降低整体效率。实际服务还需要队列策略、公平性目标和尾延迟控制。\n\nORCA 的贡献是把“批次”从固定的一组请求，变成每轮都可调整的调度单位。它让大模型在线推理更像持续运行的生产线，而非一车人一起上、一起下。理解这项技术时，要同时关注调度粒度、上下文与解码阶段，以及吞吐和用户等待时间之间的平衡。\n\n来源：[USENIX OSDI 2022：ORCA](https:\u002F\u002Fwww.usenix.org\u002Fsystem\u002Ffiles\u002Fosdi22-yu.pdf)","\u002Fuploads\u002F2026-10-17\u002F03af83bc-ae9f-4ec1-898b-2ebf94375b23.jpg",[],[],"Foundit","https:\u002F\u002Ffoundit.cn","foundit-ai-editorial",{"id":18,"name":19,"slug":20,"description":21},"6179d3b6-dc34-4483-9ded-3cd9f1b37a47","科普","abbreviation","介绍各领域新兴概念",[23,27,31],{"id":24,"name":25,"slug":26},"a2ccffe0-49b2-458b-baf6-a83a1b20443d","大语言模型","llm",{"id":28,"name":29,"slug":30},"88d2bc27-0e0f-468a-b907-2991cb97b87b","人工智能","ai",{"id":32,"name":33,"slug":34},"7c76bfc2-f80f-4ee0-a95d-27bd8708b434","技术","slug","USENIX OSDI 2022 paper","https:\u002F\u002Fwww.usenix.org\u002Fsystem\u002Ffiles\u002Fosdi22-yu.pdf","published",null,false,44,0,"2026-10-17T00:00:00.000Z","2026-10-17T03:08:58.582Z",[45,53,62],{"id":46,"type":6,"title":47,"slug":48,"summary":49,"coverUrl":50,"authorName":14,"sno":51,"publishedAt":42,"createdAt":52},"d17c55b9-6529-49b0-8175-ac61e0d6cf47","PagedAttention：大模型推理为什么也需要“内存分页”？","pagedattention-llm-kv-cache-memory","PagedAttention 把大模型推理时不断增长的 KV Cache 切分为固定大小的块，并通过映射表管理它们，减少连续分配带来的浪费。本文用图书馆书架解释它如何提升显存利用率，以及为什么它不是压缩算法或质量增强技术。","\u002Fuploads\u002F2026-10-17\u002F8a2a3d13-37cb-40c1-9855-cdcbe0f0d2f1.jpg",57,"2026-10-17T03:08:54.233Z",{"id":54,"type":6,"title":55,"slug":56,"summary":57,"coverUrl":58,"authorName":14,"sno":59,"publishedAt":60,"createdAt":61},"5ef7ce44-0678-40b4-95e0-af4fe8a6b0a7","提示词也能缓存：固定前缀为什么能省钱提速？","prompt-caching-prefix-kv-cache","Prompt caching 缓存的不是旧答案，而是模型处理重复提示词前缀时产生的中间状态。本文解释它与语义缓存的区别、为什么顺序会影响命中、如何整理 Agent 上下文，以及多租户场景中的隔离风险。","\u002Fuploads\u002F2026-09-08\u002F182822f9-05a3-4d2b-8766-bc5daf93effd.jpg",42,"2026-09-08T00:00:00.000Z","2026-09-08T03:19:26.522Z",{"id":63,"type":6,"title":64,"slug":65,"summary":66,"coverUrl":67,"authorName":14,"sno":68,"publishedAt":60,"createdAt":69},"560f8daa-d53e-49f9-b4cc-54d5f39aea71","1.58 比特大模型：参数只有 −1、0、1，为什么还能推理？","one-bit-llm-bitnet-explained","BitNet b1.58 让模型从训练阶段就使用 −1、0、1 三值权重，探索更低的内存、延迟和能耗。本文解释 1.58 bit 的由来、原生低比特与 INT4 量化的区别、零值的作用，以及为什么收益依赖硬件和运行时。","\u002Fuploads\u002F2026-09-08\u002Fab641d19-d38a-4d44-b40f-649aa1c3191a.jpg",49,"2026-09-08T03:19:29.625Z"]