fix: improve multimodal image processing and message handling #545
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…age handling - Remove unused base.ts export file - Add getMessageContent import and validation in read_chat_message middleware - Improve image processing workflow by using separate message object - Fix image content type from 'image' to 'image_url' for proper compatibility - Update image prompt templates for better descriptions - Add proper disposal handling for message transformer interceptor - Enhance text content handling for both string and complex message types
Contributor
Walkthrough移除 model/base.ts 对 Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant U as 用户
participant MW as Chat 中间件
participant Bot as 下游处理
U->>MW: 发送消息
MW->>MW: transformedMessage.content 判空
MW->>MW: getMessageContent(transformedMessage.content).trim() 判空
alt 内容为空
MW-->>Bot: 停止后续处理
else 有内容
MW-->>Bot: 继续后续流程
end
sequenceDiagram
autonumber
participant Ctx as Koishi 上下文
participant ISP as Image Service 插件
participant Img as 图片来源
participant Model as 模型
Ctx->>ISP: 注册消息拦截（返回 disposable）
note right of ISP: ctx.effect 管理清理
Img-->>ISP: 命中图片消息
ISP->>ISP: 创建 fakeMessage(content: [])
ISP->>ISP: addImageToContent(fakeMessage, base64)
ISP->>Model: processImageWithModel(fakeMessage)
note over Model: 内联构造 HumanMessage；输出调试日志
alt 生成结果
Model-->>ISP: 文本结果
ISP->>ISP: addTextToContent(originalMessage, "\n\n" + result)
else 无结果
Model-->>ISP: 无输出
end
ISP-->>Ctx: 继续后续中间件/处理
Estimated code review effort🎯 4 (复杂) | ⏱️ ~45 minutes Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://waps.l3s.uni-hannover.de/live/im_/https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
packages/core/src/middlewares/chat/read_chat_message.ts (2)
120-124: 修复文件名生成：重复空合并、自增花括号、多数情况下缺失扩展名
- 左右两侧都取
filename无意义。- 模板串多出一个
}。- 未拼接推断的扩展名，可能影响下游存储/识别。
应用如下补丁：
- let fileName = - element.attrs['filename'] ?? element.attrs['filename'] + let fileName = element.attrs['filename'] as string | undefined @@ - if (fileName == null || fileName.length > 50) { - fileName = `${await hashString(url, 8)}}` - } + if (!fileName || fileName.length > 50) { + fileName = `${await hashString(url, 8)}.${ext}` + }
169-174: 音频拦截器在 content 为数组时会破坏结构
message.content += content假定为字符串，若为数组会被隐式转为字符串，破坏消息结构。应复用已有的追加函数。应用如下补丁：
- const content = await ctx.sst.audio2text(session) - logger.debug(`audio2text: ${content}`) - message.content += content + const content = await ctx.sst.audio2text(session) + logger.debug(`audio2text: ${content}`) + addMessageContent(message, content)
🧹 Nitpick comments (5)
packages/core/src/middlewares/chat/read_chat_message.ts (2)
188-195: 为外部 HTTP 拉取增加超时以避免请求悬挂建议添加合理超时（例如 10s）。如
ctx.http兼容 axios 选项，可直接使用timeout。应用如下补丁（请确认
timeout配置项在当前 Koishi 版本有效）：const response = await ctx.http(url, { responseType: 'arraybuffer', method: 'get', headers: { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36' - } + }, + timeout: 10000 })
97-101: 文案小修（可选）“安装了”语法更自然，便于用户理解。
应用如下补丁：
-If you are install chatluna-image-service plugin, please ignore this warning. +If you have installed the chatluna-image-service plugin, you can ignore this warning.packages/image-service/src/index.ts (3)
160-161: 替换console.log为框架日志与其他日志保持一致，便于级别控制与收集。
应用如下补丁：
- console.log(images) + logger.debug('extracted images', images)
130-137: 为图片拉取增加超时同上，避免外部请求悬挂。
应用如下补丁（请确认超时配置项有效）：
const response = await ctx.http(url, { responseType: 'arraybuffer', method: 'get', headers: { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36' - } + }, + timeout: 10000 })
122-152: （可选）根据响应 Content-Type 推断扩展名当前仅从 URL 推断，遇到无扩展名/带查询串的 URL 可能退化为 jpeg。可优先使用
response.headers['content-type']映射为png/jpeg。如需我补一个 MIME→扩展名的小映射表，请告知。
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
packages/image-service/package.jsonis excluded by!**/*.json
📒 Files selected for processing (3)
packages/core/src/llm-core/model/base.ts(0 hunks)packages/core/src/middlewares/chat/read_chat_message.ts(3 hunks)packages/image-service/src/index.ts(6 hunks)
💤 Files with no reviewable changes (1)
- packages/core/src/llm-core/model/base.ts
🧰 Additional context used
🧬 Code graph analysis (2)
packages/core/src/middlewares/chat/read_chat_message.ts (1)
packages/core/src/utils/string.ts (1)
getMessageContent(91-107)
packages/image-service/src/index.ts (1)
packages/core/src/types.ts (1)
Message(53-67)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: build
- GitHub Check: lint
- GitHub Check: build
🔇 Additional comments (5)
packages/core/src/middlewares/chat/read_chat_message.ts (2)
214-221: 改用image_url结构推图 LGTM与下游
isMessageContentImageUrl对齐，结构正确。
238-252: 文本追加逻辑兼容字符串与复合消息 LGTM能够在混合内容场景下稳健追加文本。
packages/image-service/src/index.ts (3)
35-70: 使用“伪消息”承载图片并在完成后回填文本的设计合理不污染原消息，且通过
ctx.effect(() => disposable)正确清理拦截器。
184-191:image_url结构对接 LangChain/下游检测函数 LGTM类型与提取逻辑一致，兼容性良好。
193-210: 文本回填策略 LGTM优先合并末尾文本块，避免碎片化。
Change message content validation from OR to AND condition and add trim() to properly handle empty messages with whitespace content.
This was referenced Nov 30, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Changes
getMessageContentvalidation in read_chat_message middleware