How to apply the eye Ball Rotation using ARKit in iOS from FaceTracking

Question

I am using ARKit's ARFaceTrackingConfiguration to track the facial blendshapes along with left and right Eye Transforms. I am exporting this data into json and apply this data on 3d model ( which preconfigured shape keys, eye nodes). I was able to apply the blend shape data, but I got struck at how to apply the eye rotations. am getting leftEyeTransform, rightEyeTransform which is simd_float4*4 from FaceAnchor.

Here how to apply the rotation on eye nodes from the transform values.I believe for eyes, it is enough to apply the rotation.

I have tried with the below to get the orientation from eyeTransforms:

Method 1:

let faceNode = SCNNode()
faceNode.simdTransform = eyeTransform
let vector = faceNode.eulerAngles
eyeLeftNode.eulerAngles = vector

Method:2

let faceNode = SCNNode()
faceNode.simdTransform = eyeTransform
let rotation = vector_float3(faceNode.orientation.x,faceNode.orientation.y,faceNode.orientaton.z)
let yaw = (rotation.y)
let pitch = (rotation.x)
let roll = (rotation.z)
let vector = SCNVector3(pitch, yaw, roll)

eyeLeftNode.eulerAngles = vector

Method: 3

let simd_quatf = simd_quaternion(eyeTransform)
let vector = SCNVector3(simd_quatf.axis.x,simd_quatf.axis.y,simd_quatf.axis.z)

eyeLeftNode.eulerAngles = vector

None of the ways are working. I am not able to figure out the actual problem on how to rotate the eyeBalls. Can you please tell me how to do this

Thanks, Chaitanya

Answer 1

I use the following two extensions in my apps for simd_float4x4 translation and orientation components if that's all you need:

extension float4x4 {
    var translation: SIMD3<Float> {
        let translation = columns.3
        return SIMD3<Float>(translation.x, translation.y, translation.z)
    }

    /**
     Factors out the orientation component of the transform.
    */
    var orientation: simd_quatf {
        return simd_quaternion(self)
    }
}