Ovi: video and audio generation model

[kabachuha]

Ovi is a new model from Character AI trained to generate audio and video at the same time.

It consists of Wan2.2-5b and MMAudio-5b hybrid, with a 1b bridge.

https://github.com/character-ai/Ovi

It would be nice to try it out and maybe integrate into the wrapper (some cross kijai node packs interaction?)

[RuneGjerde]

that looks interesting for sure ;-) and even if its a 5B model, there are ways with 2nd pass that can get high quality output.
Plus they mention they will release both a higher quality model, and longer generation later on.

some more examples here
https://aaxwaz.github.io/Ovi/

[rkfg]

The model is 11B and it's really good. I ran some tests, it takes 28 GB in its current unquantized form, about 3.8 s/it on my 5090. 30 steps are enough. Supports multiple languages (English is best supported of course), both I2V and T2V are solid.

tmpjrjow6ax.mp4

tmp6qlrg50h.mp4

[Rudra-ai-coder]

@kabachuha yeah this is needed

[aaxwaz]

It will be great for us if Ovi could be integrated into Comfy! Thanks a lot

[Rudra-ai-coder]

@aaxwaz do you have any plans for the Wan2.2 14b models and better audio encoder?

[kabachuha]

@aaxwaz Would you, as the authors, able to assist in the implementation into the plugin? Kijai is almost a sole maintainer, and it's hard to keep so many new models on one's own back. As a return, it will make Ovi much more accessible and popular

[aaxwaz]

@Rudra-ai-coder for now we have plans to do a few things - higher resolution and better quality data sft to further improve, distillation for faster inference (we estimated could speed up at least 2x-5x more), or even longer video generation.

For 14b models, we do not have such plan yet, as the architecture and training data prepare needs to be redone. I think the above few should have higher ROI for near term

[aaxwaz]

@kabachuha We (me, chetwin and Calder) are happy to help out as long as it could benefit the community! Would be great to know a bit about where we could start and the types of things we need to do.

[snicolast]

working on it... I ported IndexTTS 2, and am trying with this one... it will take a while, very early stage atm... no promises tho!

[Rudra-ai-coder]

working on it... I ported IndexTTS 2, and am trying with this one... it will take a while, very early stage atm... no promises tho!

i would suggest connect with @kijai and @kabachuha and the maintainers by now they know how to write in such a way that is optimized

[kabachuha]

@snicolast It's not a Comfy-esque approach :)
In Comfy/KJ, as many things as possible should be modular.
E.g. the main part should be executable by the existing sampler node, while the rest of the args are provided in it with "emdeds/args" params. The models (VAE/MMAudio/Model) need to be separated, so offloading for low vram is convenient to set up

KJ integration leverages the existing samplers and acceleration methods, not only ones within the Ovi repository

[kabachuha]

ComfyUI-MMAudio https://github.com/kijai/ComfyUI-MMAudio

[Rudra-ai-coder]

@aaxwaz the model is just wooooowwwww, you guys are going toe to toe with sora2 and veo 3, just tried from your repo, i have one request next release, please give us a features to give a reference audio for the speaker, but when it comes to the I2v, this models is nuts @kabachuha when can we get in comfy brooo, so that we can tweak maybe with other LoRAs

[snicolast]

Thanks for the message, @kabachuha. Yeah, I know... this is just my own implementation. I’m mainly building it for myself, and if it works well, I’ll share it later. I prefer to stay independent from the wrapper.

A bit more detail: I got it working for now, and added an attention selector, so it doesn’t rely solely on flash_att. I’m currently running it with sage (and sdpa is available too), and the performance on my system is identical to the original repo.

I’ll also enable WAN VAE and model selectors from the official ComfyUI instead of the OVI loader, so users can work with existing models without having duplicates around. For MMAudio, I’m keeping the auto-download for now since not everyone will have those checkpoints locally. I can add a proper check later on.