[Fix] Optimize GIF frame extraction with incremental decoding #637
Conversation
…coding - Implement incremental canvas decoding to avoid redundant frame processing - Handle GIF disposal methods (0, 1, 2, 3) correctly for differential frames - Track last decoded frame position to minimize duplicate decoding work - Add GIF context text hint for better model understanding - Suppress duplicate GIF warnings when image-service plugin is installed - Increase middleware priority to 100 for proper execution order Performance improvement: For extracting frames [0, 10, 20], now decodes 20 frames total instead of 30 frames with the previous implementation.
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughGIF 警告改为仅在未检测到 image-service 插件时发出；image-service 的 GIF 拦截流在 fakeMessage 前注入说明文本并传入额外参数；GIF 帧提取实现增量画布解码；两个适配器的构造函数泛型插件类型被具体化。 Changes
Sequence Diagram(s)sequenceDiagram
participant Chat as ChatMiddleware
participant ImageService as ImageService
participant Utils as FrameUtils
participant Canvas as CanvasBuffer
Chat->>Chat: 识别到 GIF 图像
alt image-service 插件 已安装
Chat->>ImageService: 调用 intercept 处理 GIF (arg=100)
ImageService->>ImageService: 在 fakeMessage 前注入提示文本
ImageService->>Utils: 请求提取 GIF 帧
Utils->>Canvas: 检查 lastDecodedFrame 与 hasComplexDisposal
alt 需要全量解码
Utils->>Canvas: 清空并从帧0解码至当前帧
else 增量解码
Utils->>Canvas: 从 lastDecodedFrame+1 解码至当前帧
end
Canvas-->>Utils: 返回当前画布数据（复制为 frameData）
Utils-->>ImageService: 返回帧 buffers
ImageService-->>Chat: 返回带前置文本与帧的 fakeMessage
else image-service 插件 缺失
Chat->>Chat: 发出 GIF 警告并早期返回 false
end
Estimated code review effort🎯 3 (中等) | ⏱️ ~20 分钟
Possibly related PRs
兔子的诗
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🧰 Additional context used🧬 Code graph analysis (1)packages/adapter-spark/src/requester.ts (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
🔇 Additional comments (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @dingyi222666, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request delivers a crucial upgrade to the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://waps.l3s.uni-hannover.de/live/im_/https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces significant optimizations for GIF frame extraction by implementing incremental decoding and handling disposal methods. It also fixes an issue with duplicate warnings and improves the contextual information provided to the model for GIF images. The changes in read_chat_message.ts and index.ts are solid improvements. However, the core GIF decoding logic in utils.ts has a correctness issue. While it attempts to handle disposal methods, the optimization for incremental decoding is based on a flawed assumption and can lead to corrupted frames for certain GIFs. I've left a detailed comment on this critical issue.
![coderabbitai[bot]](https://waps.l3s.uni-hannover.de/live/im_/https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
packages/core/src/middlewares/chat/read_chat_message.ts(2 hunks)packages/service-image/src/index.ts(2 hunks)packages/service-image/src/utils.ts(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-09-17T00:25:27.195Z
Learnt from: dingyi222666
Repo: ChatLunaLab/chatluna PR: 548
File: packages/core/src/llm-core/chat/app.ts:0-0
Timestamp: 2025-09-17T00:25:27.195Z
Learning: 在 ChatInterface 类中，响应式 watch 调用通过 ctx.effect() 包装来自动处理清理工作，避免内存泄漏。字段 _chain 和 _embeddings 的类型已更新为可空类型 (| undefined)，并添加 ctx.on('dispose') 处理器提供额外的清理保障。这种使用 Koishi effect 系统的方式比手动管理 stop 句柄更优雅。
Applied to files:
packages/service-image/src/index.ts
🧬 Code graph analysis (2)
packages/core/src/middlewares/chat/read_chat_message.ts (3)
packages/service-image/src/index.ts (1)
logger(17-17)packages/extension-long-memory/src/index.ts (1)
logger(9-9)packages/core/src/index.ts (1)
logger(38-38)
packages/service-image/src/index.ts (1)
packages/service-image/src/utils.ts (1)
addTextToContent(220-237)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: lint
- GitHub Check: build
🔇 Additional comments (5)
packages/core/src/middlewares/chat/read_chat_message.ts (2)
176-183: 与主代码路径保持一致的条件检查。
oldImageRead函数中的条件检查与第 127-131 行逻辑一致，确保了两个代码路径的行为统一。
126-133: Conditional logic correctly gates GIF warnings based on plugin availability.The changes properly suppress the GIF unsupported warning when
image-serviceplugin is installed by checkingctx.chatluna.getPlugin('image-service') == nullbefore logging. ThegetPluginmethod is a legitimate API from the chat service that retrieves plugins from the internal registry. Returningfalseallows the middleware chain to continue, enabling theimage-serviceplugin to handle GIF processing when available.The identical conditional patterns at lines 127 and 177 handle separate message processing paths (buffer-based and URL-based image sources) and are appropriately duplicated for clarity.
packages/service-image/src/index.ts (2)
128-131: 添加上下文提示文本有助于模型理解 GIF 内容。在帧图像之前添加描述性文本 "This is a GIF image. See the frames below:" 为 AI 模型提供了必要的上下文，有助于其正确理解后续的帧序列。
154-156: Interceptor priority 100 ensures correct GIF image handling order.The priority value (100) is higher than the core module's 'img' interceptor priority (-100, line 156 of
read_chat_message.ts), ensuring the image-service intercept runs before the core image processing logic. The insertion algorithm inmessage_transform.ts(lines 146–152) maintains interceptors in descending priority order, so higher numbers execute first.packages/service-image/src/utils.ts (1)
105-111: 正确复制画布数据避免引用问题。创建
frameData的副本是正确的做法，避免了后续帧解码时覆盖已创建图像数据的问题。
Previously checked current frame's disposal method, but GIF spec requires checking previous frames' disposal methods to determine canvas preparation. Disposal methods indicate how to handle canvas AFTER displaying a frame, not before. - Add hasComplexDisposal() helper to check disposal methods in frame range - Check frames between lastDecodedFrame and current for disposal 2/3 - Properly handle disposal 2 (restore to background) and 3 (restore to previous) - Align with GIF specification for correct differential frame rendering
Add DifyClientConfig type parameter to ChatLunaPlugin for better type safety and consistency with the parent class signature.
This change adds the SparkClientConfig generic type parameter to the ChatLunaPlugin instance in the SparkRequester constructor, ensuring type consistency with other adapters.
This PR optimizes GIF frame extraction in the image-service plugin by implementing incremental decoding to handle GIF disposal methods correctly and improve performance.
Bug Fixes
Performance Improvements
Other Changes