[BUG] The Mixamorg Avatar stays still in it's T pose.

[Ushaxoxo]

Current Behavior

The 3d avatar does not move for any input, it stays in its t pose.

Expected Behavior

The mixamorg avatar is supposed to sign.

Steps To Reproduce

No response

Example flow

paste your flow here

Environment

• Device: Macbook
• OS: MacOS
• Browser:Chrome
• Version: (eg. 13; 73)
• Other info: (eg. display resolution, ease-of-access settings)

Additional context

No response

[AmitMY]

Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw
We did not further implement this avatar movement in the translation app, but you can test it out in the Playground https://sign.mt/playground

If we were to have a better model, we would implement it.

[Ushaxoxo]

Hi,
I was wondering why we can't directly map the 3d keypoints obtained as pose file into the bone structure of the mixamorig avatar.
It will be great if you can share the Structure of the pose file

[Ushaxoxo]

Hi, I was wondering why we can't directly map the 3d keypoints obtained as pose file into the bone structure of the mixamorig avatar. It will be great if you can share the Structure of the pose file

…

On Sun, Apr 20, 2025 at 1:16 PM Amit Moryossef ***@***.***> wrote: Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw We did not further implement this avatar movement in the translation app, but you can test it out in the Playground https://sign.mt/playground If we were to have a better model, we would implement it. — Reply to this email directly, view it on GitHub <#231 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4OQDZNXKLG6M4WZF4BM3GD22NGFDAVCNFSM6AAAAAB3PELGVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJXGA2DMNRRGA> . You are receiving this because you authored the thread.Message ID: ***@***.***> *AmitMY* left a comment (sign/translate#231) <#231 (comment)> Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw We did not further implement this avatar movement in the translation app, but you can test it out in the Playground https://sign.mt/playground If we were to have a better model, we would implement it. — Reply to this email directly, view it on GitHub <#231 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4OQDZNXKLG6M4WZF4BM3GD22NGFDAVCNFSM6AAAAAB3PELGVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJXGA2DMNRRGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Ushaxoxo]

waiting for your response On Sun, Apr 20, 2025 at 9:21 PM Usha nandhini ***@***.***> wrote:

…

Hi, I was wondering why we can't directly map the 3d keypoints obtained as pose file into the bone structure of the mixamorig avatar. It will be great if you can share the Structure of the pose file On Sun, Apr 20, 2025 at 1:16 PM Amit Moryossef ***@***.***> wrote: > Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw > We did not further implement this avatar movement in the translation app, > but you can test it out in the Playground https://sign.mt/playground > > If we were to have a better model, we would implement it. > > — > Reply to this email directly, view it on GitHub > <#231 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/A4OQDZNXKLG6M4WZF4BM3GD22NGFDAVCNFSM6AAAAAB3PELGVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJXGA2DMNRRGA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > *AmitMY* left a comment (sign/translate#231) > <#231 (comment)> > > Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw > We did not further implement this avatar movement in the translation app, > but you can test it out in the Playground https://sign.mt/playground > > If we were to have a better model, we would implement it. > > — > Reply to this email directly, view it on GitHub > <#231 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/A4OQDZNXKLG6M4WZF4BM3GD22NGFDAVCNFSM6AAAAAB3PELGVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJXGA2DMNRRGA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[AmitMY]

The .pose file has a .header (you can check out the format here https://github.com/sign-language-processing/pose)
That gives you points and their 3D positions.

In theory, what you are saying is possible. I have seen people map the keypoints to a 3D mesh, but the main problem is that some specific interactions (for example, the letter M in ASL) is not fully correctly represented in MediaPipe.
For that, a model could "recover" from the errors - if it sees something strange at this or that pattern, it can learn how to recover.

I did some attempts at looking at the 3D positioning of the predicted pose, you can look at these videos:

2025-02-15.15-49-59.mp4

2025-02-15.15-54-11.mp4

[marsiliflavio]

hi,
I am adding to the discussion. I created a project for my work that moves an avatar in real time by capturing mediapipe from pc camera. It works really well and I can tell you that with mixamo this cannot be done at all.
To create an avatar that correctly interprets the pose, face and hands keypoints of mediapipe you need to work with blender (for example) and do the correct rigging of all the bones, create the meshes and determine what influences the meshes have based on the rotation or displacement of the individual bones. Mixamo's “puppets” do not support all this.
The princiapal problems established the avatar with blender and realtime motion mapping, or passing a pre-recorded motion of keypoints mediapipe are:

• jitter: can be mitigated with smoothNET by disabling that of mediapipe poses if we prerecord the movements and give them in batch to the avatar
• Z-axis of the pose: established a fixed front view and optionally blocking the rotation of the pelvis, the Z-axis shoulder joint is still not properly evaluated by mediapipe because not one camera you don't have depth and mediapipe has a 2D view of the captured frame. I am working on this since I want to use only one camera, e.g., the phone camera.
  Finally keep in mind that threejs is webgl and not opengl so it does not take full advantage of avatar rendering (glb/gltf). If you want avatar beautiful as on blender that are GPU you have to write native code that takes advantage of hardware resources.

[AmitMY]

Very cool @marsiliflavio
Do you have a demo to try?

Want to integrate it with the app? If so, I think all we need is a GLTF file for the avatar with all the bones etc (and make sure it loads with model-viewer) then your mapping function.

[marsiliflavio]

Can the email hello@sign.mt be used to talk to you? I can't find your email in the profile.
your work is super. i am part of project called HandySigns for LIS (italian sign language) translation and mostly app/web and avatar development

[ashar02]

great job @marsiliflavio
any demo to try it?

[AmitMY]

you can reach out at amit@sign.mt, however, any message related to this project, as long as not containing some secret information, i'd rather manage in github.