Avatar Pupettering with ThreeJS, ReadyPlayerMe, Kalidokit and MediaPipe

Question

I try to animate a ReadyPlayer Me avatar using ThreeJS and Kalidokit (or something else) with MediaPipe Hollisitc Pose. Here is a working JSFiddle :

https://jsfiddle.net/nxg5bp2h/1/

It almost works, but the code for Pose Estimation is all messed-up. I use the one inspired from the scripts code of Kalidokit but arms seems reversed and the legs going up ...

let  riggedPose = Kalidokit.Pose.solve(poselm3D, poselm,{runtime:'mediapipe',video: 
$('#mediapipe-video')[0]})
if (!riggedPose) { return; }

this.rigRotation('Hips', riggedPose.Hips.rotation, 0.7)
this.rigPosition('Hips', {
    x: -riggedPose.Hips.position.x, // Reverse direction
    y:  riggedPose.Hips.position.y + 1, // Add a bit of height
    z: -riggedPose.Hips.position.z // Reverse direction
}, 1, 0.07)

this.rigRotation('Chest', riggedPose.Spine, 0.25, 0.3)
this.rigRotation('Spine', riggedPose.Spine, 0.45, 0.3)

this.rigRotation('RightUpperArm', riggedPose.RightUpperArm, 1, 0.3)
this.rigRotation('RightLowerArm', riggedPose.RightLowerArm, 1, 0.3)
this.rigRotation('LeftUpperArm',  riggedPose.LeftUpperArm,  1, 0.3)
this.rigRotation('LeftLowerArm',  riggedPose.LeftLowerArm,  1, 0.3)

this.rigRotation('LeftUpperLeg',  riggedPose.LeftUpperLeg,  1, 0.3)
this.rigRotation('LeftLowerLeg',  riggedPose.LeftLowerLeg,  1, 0.3)
this.rigRotation('RightUpperLeg', riggedPose.RightUpperLeg, 1, 0.3)
this.rigRotation('RightLowerLeg', riggedPose.RightLowerLeg, 1, 0.3)

If someone can help me fix it, spot the error or to do the math ? I'm a little bit lost to understand where/why rotation are incorrect. May be all the RPM models are rigged another way ?!

Answer 1

I found that if you swap the y and z rotations then you get a better result... it's still a bit unstable but it's at least in the right proximity.

let euler = new this.THREE.Euler(rotation.x * dampener, rotation.z * dampener, rotation.y * dampener)

Answer 2

For the Arms, I would found that swapping left to right worked significantly better, they seem to be mirrored. So instead of this.rigRotation('RightUpperArm', riggedPose.RightUpperArm, 1, 0.3) I did: this.rigRotation('LeftUpperArm', riggedPose.RightUpperArm, 1, 0.3) and so on.

Also, you are trying to access the "Chest" bone on your rig with this.rigRotation('Chest', riggedPose.Spine, 0.25, 0.3), however this bone does not exist on the ReadyPlayerMe rig. The RPM rig has "shoulderLeft" and "shoulderRight" bones in the Armature instead. When computing the quaternions, I assume that this is where it breaks, since the rotations are missing rotations in the traversal of the armature which lead to all bones lower in the tree being effected.