[Feature] Add GIF animation support and improve image handling #572
Conversation
- Add getImageType function to detect image format from buffer content - Replace file extension-based detection with buffer-based detection - Add warning when GIF images are detected (not supported by most models) - Return proper MIME types for image formats (png, jpeg, gif, webp) - Fix filename generation to include proper extension
…mage handling - Add GIF frame extraction with configurable strategies (first/head/average) - Add jimp and omggif dependencies for GIF processing - Improve image type detection and logging messages - Add dedicated utils module for image processing functions - Support multi-frame GIF input for vision-capable models - Update locales with new GIF configuration options The image-service plugin now properly handles GIF images by extracting frames based on user-selected strategy, enabling better support for animated content in both vision-capable models and models requiring image descriptions.
|
Warning Rate limit exceeded@dingyi222666 has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 0 minutes and 54 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ⛔ Files ignored due to path filters (26)
📒 Files selected for processing (3)
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. Walkthrough本次变更统一并抽取了图片读取与类型识别逻辑：引入 Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant User as User
participant Core as read_chat_message
participant ImgUtils as service-image/utils
participant Index as service-image/index
participant Store as TempStorage
participant Model as ImageUnderstandModel
User->>Core: 发送含图片的消息或 URL
Core->>ImgUtils: readImage(url/data)
alt 读取成功
ImgUtils-->>Core: { buffer, ext, base64Source }
alt ext == image/gif
Core->>Index: 转交到 service-image/index 进行 GIF 路径
Index->>Model: 检查 ImageInput 能力
alt 支持 ImageInput
Index->>ImgUtils: parseGifToFrames(buffer, config)
ImgUtils-->>Index: [frame dataURLs...]
Index->>Core: 注入每帧为消息内容（可能提前返回）
else 不支持
Index-->>Core: 记录并按回退逻辑处理（提示用户）
end
else 非 GIF
Core->>Store: 可选保存临时文件（带扩展名）
Store-->>Core: 临时 URL
Core->>Index: processImageWithModel(...) 或 直接 addImageToContent
Index-->>Core: 更新后的消息内容
end
else 读取失败
Core->>Core: 记录错误（截断 URL），回退旧流程或提示
end
sequenceDiagram
autonumber
participant Incoming as Message Flow
participant Utils as service-image/utils
participant Model as ChatLuna (imageUnderstandModel)
Incoming->>Utils: extractImages(content)
alt 有图片
Utils->>Utils: 构建 prompt + image content
Utils->>Model: processImageWithModel(prompt, images)
alt 成功
Model-->>Utils: 文本替换结果
Utils->>Incoming: addTextToContent / addImageToContent
else 失败
Utils-->>Incoming: 返回 null 或原始内容
end
else 无图片
Incoming-->>Utils: 跳过
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @dingyi222666, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly upgrades the image processing capabilities by introducing comprehensive support for GIF animations. It allows the system to intelligently extract and provide multiple frames of a GIF to vision models, improving their ability to interpret dynamic content. The changes also include a major refactoring of image utility functions, leading to a more modular and maintainable codebase, alongside refined image type detection and error handling. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![codefactor-io[bot]](https://waps.l3s.uni-hannover.de/live/im_/https://avatars.githubusercontent.com/in/25603?s=60&v=4){kind=small}
![gemini-code-assist[bot]](https://waps.l3s.uni-hannover.de/live/im_/https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces significant enhancements, including GIF animation support and refactoring of image handling logic. The changes are well-structured, moving utility functions into a dedicated module. However, I've found a few critical issues, including an incorrect import and a bug in filename generation that need to be addressed. There are also some opportunities to improve logic completeness for vision-capable models and reduce code duplication.
Co-authored-by: codefactor-io[bot] <47775046+codefactor-io[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…hModel Removed debug console.log statement that was logging extracted images array. This change cleans up the code and removes unnecessary output during image processing.
- Remove extra closing brace in logger.debug call - Add console.log to output extracted images in processImageWithModel function
…hModel Removed debug console.log statement that was logging extracted images array. This change cleans up the code and removes unnecessary output during image processing.
![coderabbitai[bot]](https://waps.l3s.uni-hannover.de/live/im_/https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (2)
packages/service-image/src/utils.ts (2)
137-137: 移除调试输出，改用 logger这里的
console.log属于调试遗留，正式代码应改用现有的logger.debug/logger.info，避免污染标准输出。
123-129: 为未识别的图片类型提供 MIME 兜底当
getImageType返回undefined（例如 BMP、HEIC 等格式）时，这里会构造出data:undefined;base64,...，导致下游模型或浏览器无法识别图片，功能直接失效。请在无法识别时给出可靠兜底（如默认image/jpeg）并记录告警，确保返回值始终是合法的 data URL。- const ext = getImageType(buffer) - - return { - base64Source: `data:${ext};base64,${base64}`, - buffer, - ext - } + const detectedExt = getImageType(buffer) + const ext = detectedExt ?? 'image/jpeg' + if (!detectedExt) { + logger.warn(`未能识别图片 MIME 类型，默认使用 image/jpeg：${url}`) + } + + return { + base64Source: `data:${ext};base64,${base64}`, + buffer, + ext + }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (4)
packages/core/package.jsonis excluded by!**/*.jsonpackages/service-image/package.jsonis excluded by!**/*.jsonpackages/service-image/src/locales/en-US.schema.ymlis excluded by!**/*.ymlpackages/service-image/src/locales/zh-CN.schema.ymlis excluded by!**/*.yml
📒 Files selected for processing (4)
packages/core/src/middlewares/chat/read_chat_message.ts(5 hunks)packages/core/src/utils/string.ts(1 hunks)packages/service-image/src/index.ts(5 hunks)packages/service-image/src/utils.ts(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (3)
packages/core/src/middlewares/chat/read_chat_message.ts (2)
packages/service-image/src/utils.ts (1)
addImageToContent(158-173)packages/core/src/utils/string.ts (2)
hashString(336-349)getImageType(91-113)
packages/service-image/src/index.ts (1)
packages/service-image/src/utils.ts (4)
readImage(98-130)parseGifToFrames(87-96)addImageToContent(158-173)processImageWithModel(131-156)
packages/service-image/src/utils.ts (4)
packages/service-image/src/index.ts (3)
logger(17-17)Config(149-155)Config(157-180)packages/core/src/services/types.ts (1)
Context(21-23)packages/core/src/utils/string.ts (3)
getImageType(91-113)getMessageContent(115-131)isMessageContentImageUrl(39-43)packages/core/src/types.ts (1)
Message(53-67)
🪛 GitHub Check: CodeFactor
packages/core/src/middlewares/chat/read_chat_message.ts
[warning] 134-134: packages/core/src/middlewares/chat/read_chat_message.ts#L134
Replace ⏎················Saving·image·as·temp·file:·${fileName}⏎············ with Saving·image·as·temp·file:·${fileName} (prettier/prettier)
[warning] 105-105: packages/core/src/middlewares/chat/read_chat_message.ts#L105
Insert ⏎··············· (prettier/prettier)
[notice] 99-99: packages/core/src/middlewares/chat/read_chat_message.ts#L99
This line has a length of 192. Maximum allowed is 160. (max-len)
[warning] 176-176: packages/core/src/middlewares/chat/read_chat_message.ts#L176
Insert ⏎··············· (prettier/prettier)
packages/service-image/src/index.ts
[notice] 11-11: packages/service-image/src/index.ts#L11
Member 'addImageToContent' of the import declaration should be sorted alphabetically. (sort-imports)
packages/service-image/src/utils.ts
[warning] 67-67: packages/service-image/src/utils.ts#L67
Delete ⏎ (prettier/prettier)
[warning] 74-74: packages/service-image/src/utils.ts#L74
Replace ·data:·Buffer.from(frameRGBA),·width,·height with ⏎················data:·Buffer.from(frameRGBA),⏎················width,⏎················height⏎··········· (prettier/prettier)
[warning] 56-56: packages/service-image/src/utils.ts#L56
Replace {·length:·totalFrames·},·(_,·i)·=>·i with ⏎························{·length:·totalFrames·},⏎························(_,·i)·=>·i⏎···················· (prettier/prettier)
[notice] 105-105: packages/service-image/src/utils.ts#L105
Expected property shorthand. (object-shorthand)
[warning] 39-39: packages/service-image/src/utils.ts#L39
Delete ⏎ (prettier/prettier)
[notice] 4-4: packages/service-image/src/utils.ts#L4
Member 'HumanMessage' of the import declaration should be sorted alphabetically. (sort-imports)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: build
- GitHub Check: lint
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (4)
packages/service-image/src/index.ts (2)
54-55: 避免向 readImage 传入 undefined部分平台只提供
element.attrs.url，element.attrs.src可能为undefined。现在直接传给readImage会在内部调用startsWith时抛出 TypeError，导致图片整体无法处理。与上一轮相同问题，请统一使用已归一化的url。- imageData = await readImage(ctx, element.attrs.src) + imageData = await readImage(ctx, url)
48-77: 为支持 Vision 的模型补充非 GIF 图像路径当前分支仅在 GIF 场景返回，普通 PNG/JPEG 会继续走描述模型流程，导致具备图像输入能力的模型拿不到原图。这和先前 reviewer 的反馈一致。请在同一分支里为非 GIF 情况将原图加入
message并提前返回。if ( - parsedModelInfo.value != null && - parsedModelInfo.value.capabilities.includes( + imageCapableModel != null && + imageCapableModel.capabilities.includes( ModelCapabilities.ImageInput ) ) { - imageData = await readImage(ctx, url) + imageData = await readImage(ctx, url) if (imageData.ext === 'image/gif') { logger.debug(`image url: ${url.substring(0, 50)}...`) // Parse GIF and add multiple frames for models that support image input const frames = await parseGifToFrames( imageData.buffer, { strategy: config.gifStrategy, frameCount: config.gifFrameCount } ) logger.debug( `Extracted ${frames.length} frames from GIF` ) for (const frame of frames) { addImageToContent(message, frame) } return true } + + addImageToContent(message, imageData.base64Source) + return true }packages/service-image/src/utils.ts (2)
53-63: 平均抽帧需覆盖首尾帧
totalFrames / count的取整会在无法整除时漏掉末帧，例如 3 帧取 2 帧得到[0,1]。需要改用基于totalFrames - 1的等距采样，并单独处理count === 1，与上一条评论保持一致，以保证始终包含首尾。case 'average': { const count = Math.min(config.frameCount, totalFrames) if (count >= totalFrames) { frameIndices = Array.from({ length: totalFrames }, (_, i) => i) + } else if (count === 1) { + frameIndices = [0] } else { - const step = totalFrames / count - frameIndices = Array.from({ length: count }, (_, i) => - Math.floor(i * step) - ) + const step = (totalFrames - 1) / (count - 1) + frameIndices = Array.from({ length: count }, (_, i) => + Math.min( + totalFrames - 1, + Math.round(i * step) + ) + ) } break }
99-129: 为未知类型提供 MIME 类型兜底
getImageType可能返回undefined（格式识别失败或未支持），此时生成的data:${ext};base64,...会变成data:undefined;...，无法被下游模型识别。请在两个返回路径都加上 MIME 兜底，与此前建议保持一致。- const ext = getImageType(buffer) + const ext = getImageType(buffer) ?? 'image/jpeg' return { base64Source: url, - buffer: buffer, + buffer, ext } } @@ - const ext = getImageType(buffer) + const ext = getImageType(buffer) ?? 'image/jpeg' return { base64Source: `data:${ext};base64,${base64}`, buffer, ext }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
packages/service-image/src/index.ts(5 hunks)packages/service-image/src/utils.ts(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
packages/service-image/src/utils.ts (3)
packages/service-image/src/index.ts (3)
logger(17-17)Config(149-155)Config(157-180)packages/core/src/utils/string.ts (3)
getImageType(91-113)getMessageContent(115-131)isMessageContentImageUrl(39-43)packages/core/src/llm-core/platform/model.ts (1)
ChatLunaChatModel(96-572)
packages/service-image/src/index.ts (2)
packages/core/src/services/chat.ts (1)
platform(315-317)packages/service-image/src/utils.ts (4)
readImage(98-130)parseGifToFrames(87-96)addImageToContent(158-173)processImageWithModel(131-156)
🪛 GitHub Check: CodeFactor
packages/service-image/src/utils.ts
[warning] 67-67: packages/service-image/src/utils.ts#L67
Delete ⏎ (prettier/prettier)
[warning] 74-74: packages/service-image/src/utils.ts#L74
Replace ·data:·Buffer.from(frameRGBA),·width,·height with ⏎················data:·Buffer.from(frameRGBA),⏎················width,⏎················height⏎··········· (prettier/prettier)
[warning] 56-56: packages/service-image/src/utils.ts#L56
Replace {·length:·totalFrames·},·(_,·i)·=>·i with ⏎························{·length:·totalFrames·},⏎························(_,·i)·=>·i⏎···················· (prettier/prettier)
[notice] 105-105: packages/service-image/src/utils.ts#L105
Expected property shorthand. (object-shorthand)
[warning] 39-39: packages/service-image/src/utils.ts#L39
Delete ⏎ (prettier/prettier)
[notice] 4-4: packages/service-image/src/utils.ts#L4
Member 'HumanMessage' of the import declaration should be sorted alphabetically. (sort-imports)
packages/service-image/src/index.ts
[notice] 11-11: packages/service-image/src/index.ts#L11
Member 'addImageToContent' of the import declaration should be sorted alphabetically. (sort-imports)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: build
- GitHub Check: lint
…e sampling - Extract clean file extension from MIME type (strip "image/" prefix) - Fix filename template to remove extra '}' character - Fix average frame sampling to include final frame using proper span calculation - Handle count === 1 case to avoid division by zero
Reorder import statements in index.ts for better readability. Fix optional chaining when accessing parsedModelInfo value. Add debug log message when plugin is loaded.
Update peerDependencies across all adapter and extension packages to align with the new core version. Also update image-service version to 1.3.0-alpha.0 for consistency.
This PR adds comprehensive GIF animation support and improves image handling across the image processing pipeline.
New Features
utils.tsmodule with reusable image processing functionsBug Fixes
Other Changes
jimpandomggifdependencies for GIF processing