Let's explore the world modeling potentials on MindSpore ;)

MindSpore implementation for Janus-Pro training/inference is now released! Supporting both multimodal understanding and visual generation on Ascend NPU. Decoupling visual encoding generation/understanding-specific tasks surely bring Omni-capability. Details can be found here.

MVDream is a diffusion model that is able to generate consistent multiview images from a given text prompt. It shows that learning from both 2D and 3D data, a multiview diffusion model can achieve the generalizability of 2D diffusion models and the consistency of 3D renderings. Details can be found here

These videos are rendered from the trained 3D implicit field in our MVDream model. Color meshes are extracted with the script MVDream-threestudio/extract_color_mesh.py.

We support instantmesh for the 3D mesh generation using the multiview images extracted from the sv3d pipeline.

Using the multiview images input from 3D mesh extracted from the sv3d pipeline, we extracted 3D meshes as below. Please kindly find the input illustrated by following the link to the sv3d pipeline below.

The illustrations here are better viewed in viewers than with HTML support (e.g., the vscode built-in viewer).

Output Multiview Images (21x576x576)

A camera-guided diffusion model that can generate the multiview snippet of a given image! Details can be found here.

To install MindONE v0.3.0, please install MindSpore 2.5.0 and run pip install mindone

Alternatively, to install the latest version from the master branch, please run.

git clone https://github.com/mindspore-lab/mindone.git
cd mindone
pip install -e .

We support state-of-the-art diffusion models for generating images, audio, and video. Let's get started using Stable Diffusion 3 as an example.

Hello MindSpore from Stable Diffusion 3!

import mindspore
from mindone.diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers",
    mindspore_dtype=mindspore.float16,
)
prompt = "A cat holding a sign that says 'Hello MindSpore'"
image = pipe(prompt)[0][0]
image.save("sd3.png")

• mindone diffusers is under active development, most tasks were tested with mindspore 2.5.0 on Ascend Atlas 800T A2 machines.
• compatibale with hf diffusers 0.32.2