[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$fzKc3YcWxQKY-3F-aLn3ykHjzUZgbqfllgMZ83DOJPoo":3},{"item":4,"related":43},{"id":5,"type":6,"title":7,"slug":8,"summary":9,"body":10,"coverUrl":11,"productScreenshots":12,"productLinks":13,"authorName":14,"authorUrl":15,"authorSubject":16,"category":17,"tags":22,"sourceLabel":35,"sourceName":35,"sourceUrl":35,"status":36,"seoTitle":35,"seoDescription":35,"canonicalUrl":35,"isFeatured":37,"sno":38,"sortOrder":39,"publishedAt":40,"updatedAt":41,"createdAt":42},"3caa3ab7-a584-4c94-8196-e3d1bd420d5f","article","多模态大模型：让 AI 不只读文字，还能看懂图、听懂话","multimodal-llm-vision-speech","GPT-4V、Gemini、Qwen-VL 能看图听声。本文用最直白的方式讲清多模态的底层思路——把图\u002F语音编码成和文字同一向量空间的 token 再统一推理，给出多模态接口最小调用，以及成本、幻觉、隐私等边界。","早期大模型只吃文字。现在 GPT-4V、Gemini、Qwen-VL 这类多模态模型已经能看图说话、能听语音、能读表格。多模态让 AI 从「文本处理器」变成「能感知世界」的助手——你甩一张截图、一段录音、一版设计稿，它都能接得住。\n\n## 背景：为什么要多模态\n\n真实世界的信息大量是非文本的：产品截图、监控画面、会议录音、扫描合同。只处理文字的 AI，面对用户发来的图片和语音就直接「失明失聪」。把感知模态补齐，AI 才能真正嵌入工作流。\n\n## 怎么做到的\n\n核心思路一句话：把图、语音先编码成和文字同一个「向量空间」的 token，再和文本拼在一起喂给同一个 transformer。模型不区来源，统一当成 token 序列处理。\n\n- **视觉**：用视觉编码器（如 ViT）把图片切成小块（patch），逐块编码成 token。\n- **音频**：把语音转成频谱图，再按类似视觉的方式编码。\n- 之后文本、图像、音频 token 混在一起进入 LLM，统一推理。\n\n```mermaid\nflowchart LR\n    A[图片] --> B[视觉编码器]\n    C[语音] --> D[音频编码器]\n    E[文本] --> F[词嵌入]\n    B --> G[统一向量 token]\n    D --> G\n    F --> G\n    G --> H[同一个 LLM]\n    H --> I[回答]\n```\n\n## 一个最小可运行的例子\n\n以多模态接口为例，把图片 URL 作为「图片类型」内容传给模型：\n\n```python\nfrom openai import OpenAI\nclient = OpenAI()\n\nresp = client.chat.completions.create(\n    model=\"gpt-4o\",\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"这张图里有什么？\"},\n            {\"type\": \"image_url\", \"image_url\": {\"url\": \"https:\u002F\u002Fexample.com\u002Fcat.png\"}},\n        ],\n    }],\n)\nprint(resp.choices[0].message.content)\n```\n\n## 取舍与边界\n\n- **成本高**：多模态输入 token 更贵，图片按分辨率切片计费，长视频更是烧钱。\n- **幻觉更隐蔽**：模型可能「看错」图里的细节（比如把 3 看成 8），且错误无法像文字那样逐字核对。\n- **延迟更大**：编码 + 超长上下文，响应比纯文本慢一截。\n- **安全与隐私**：能看图也意味着能读敏感截图，上传前要做好脱敏。\n\n## Tips\n- 用户发图\u002F发文件的场景，直接上多模态模型，别再自己写 OCR\u002F预处理硬抠。\n- 图片分辨率按需给，不必盲目传原图，能省不少 token。\n- 关键事实（数字、名称）让模型同时给「出处」，降低看错风险。\n- 涉及隐私的图片，先在端上脱敏再上传。\n- 把多模态当作「感知层」，决策和结构化仍交给后面的逻辑。","https:\u002F\u002Foxqtewbrpuiouqqjrvdv.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fpublic-media\u002F2026-07-20\u002F2e1a1ea9-fe4d-4fa5-a096-52fd673f665e.jpg",[],[],"Foundit AI","https:\u002F\u002Ffoundit.cn","f39339b1-aaa6-4e86-b0c2-a6e6a21113b5",{"id":18,"name":19,"slug":20,"description":21},"6179d3b6-dc34-4483-9ded-3cd9f1b37a47","科普","abbreviation","介绍各领域新兴概念",[23,27,31],{"id":24,"name":25,"slug":26},"88d2bc27-0e0f-468a-b907-2991cb97b87b","人工智能","ai",{"id":28,"name":29,"slug":30},"4c2bbea6-eab7-40a8-8447-1de478ff7749","分析","analyse",{"id":32,"name":33,"slug":34},"7c76bfc2-f80f-4ee0-a95d-27bd8708b434","技术","slug",null,"published",false,79,0,"2026-07-20T00:00:00.000Z","2026-07-20T10:32:20.420Z","2026-07-20T10:23:21.024Z",[44,54,63],{"id":45,"type":6,"title":46,"slug":47,"summary":48,"coverUrl":49,"authorName":50,"sno":51,"publishedAt":52,"createdAt":53},"f00e5274-8e0b-40fe-af3b-b6a8d0183b9c","一张贴纸就能骗过视觉 AI？对抗样本不是魔法","adversarial-examples-fool-visual-ai","人眼仍能认出的物体，经过精心设计的微小扰动或标记后，机器却可能改变判断。这类输入称为对抗样本。本文解释攻击者如何利用模型的决策边界，现实攻击为何比实验更难，以及为什么目前不存在一劳永逸的防御。","\u002Fuploads\u002F2026-09-08\u002F151f887b-c5f1-4647-b369-cdda8a1e1b4f.jpg","Foundit",50,"2026-09-03T00:00:00.000Z","2026-08-14T03:06:09.765Z",{"id":55,"type":6,"title":56,"slug":57,"summary":58,"coverUrl":59,"authorName":50,"sno":60,"publishedAt":61,"createdAt":62},"305492d4-c146-456a-b341-31140ab9cafd","天气预报不再一格一格算空气，AI 是怎么预测风暴的？","how-ai-weather-forecasting-works","传统数值预报依据物理方程推进大气状态，AI 天气模型则从历史观测与再分析数据中学习状态如何演变。本文以 GraphCast 为例，解释图神经网络如何快速预测全球天气、它与传统方法如何协作，以及极端天气仍有哪些难点。","\u002Fuploads\u002F2026-08-16\u002Fa8b5ddad-d6db-4d12-a159-87a6c5309082.jpg",60,"2026-08-16T00:00:00.000Z","2026-08-14T03:06:12.102Z",{"id":64,"type":6,"title":65,"slug":66,"summary":67,"coverUrl":68,"authorName":50,"sno":60,"publishedAt":69,"createdAt":70},"7f962194-e0c6-4072-9b50-c70054beb0e5","同一个问题问三遍，AI 为什么会给出三个答案？","why-ai-gives-different-answers-sampling","大模型每次回答都在从候选词中继续选择，而不是从数据库里取出一段固定文字。温度、Top-p 和随机采样共同决定回答更稳定还是更有变化。本文用抽签与岔路的比喻，解释 AI 的随机性从哪里来，以及什么时候应该追求一致。","\u002Fuploads\u002F2026-08-14\u002Fb322d338-b5c0-48c7-addf-fbd91421e316.jpg","2026-08-14T00:00:00.000Z","2026-08-14T03:06:07.351Z"]