Aligning ARFaceAnchor with SpriteKit overlay

Question

I'm trying to calculate SpriteKit overlay content position (not just overlaying visual content) over specific geometry points ARFaceGeometry/ARFaceAnchor.

I'm using SCNSceneRenderer.projectPoint from the calculated world coordinate, but the result is y inverted and not aligned to the camera image:

let vertex4 = vector_float4(0, 0, 0, 1)
let modelMatrix = faceAnchor.transform
let world_vertex4 = simd_mul(modelMatrix, vertex4)
let pt3 = SCNVector3(x: Float(world_vertex4.x),
                     y: Float(world_vertex4.y),
                     z: Float(world_vertex4.z))
let sprite_pt = renderer.projectPoint(pt3)

// To visualize sprite_pt
let dot = SKSpriteNode(imageNamed: "dot")
dot.size = CGSize(width: 7, height: 7)
dot.position = CGPoint(x: CGFloat(sprite_pt.x), 
                       y: CGFloat(sprite_pt.y))
overlayScene.addChild(dot)

Answer 1

Found the transformation that works using camera.projectPoint instead of the renderer.projectPoint.

To scale the points correctly on the spritekit: set scaleMode=.aspectFill

I updated https://github.com/AnsonT/ARFaceSpriteKitMapping to demo this.

    guard let faceAnchor = anchor as? ARFaceAnchor,
        let camera = sceneView.session.currentFrame?.camera,
        let sie = overlayScene?.size
    else { return }

    let modelMatrix = faceAnchor.transform

    let vertices = faceAnchor.geometry.vertices

    for vertex in vertices {
        let vertex4 = vector_float4(vertex.x, vertex.y, vertex.z, 1)
        let world_vertex4 = simd_mul(modelMatrix, vertex4)
        let world_vector3 = simd_float3(x: world_vertex4.x, y: world_vertex4.y, z: world_vertex4.z)

        let pt = camera.projectPoint(world_vector3, orientation: .portrait, viewportSize: size)

        let dot = SKSpriteNode(imageNamed: "dot")
        dot.size = CGSize(width: 7, height: 7)
        dot.position = CGPoint(x: CGFloat(pt.x), y: size.height - CGFloat(pt.y))
        overlayScene?.addChild(dot)

    }

Answer 2

In my experience, the screen coordinates given by ARKit's projectPoint function are directly usable when drawing to, for example, a CALayer. This means they follow iOS coordinates as described here, where the origin is in the upper left and y is inverted.

SpriteKit has its own coordinate system:

The unit coordinate system places the origin at the bottom left corner of the frame and (1,1) at the top right corner of the frame. A sprite’s anchor point defaults to (0.5,0.5), which corresponds to the center of the frame.

Finally, SKNodes are placed in an SKScene which has its origin on the bottom left. You should ensure that your SKScene is the same size as your actual view, or else the origin may not be at the bottom left of the view and thus your positioning of the node from view coordinates my be incorrect. The answer to this question may help, in particular checking the AspectFit or AspectFill of your view to ensure your scene is being scaled down.

The Scene's origin is in the bottom left and depending on your scene size and scaling it may be off screen. This is where 0,0 is. So every child you add will start there and work its way right and up based on position. A SKSpriteNode has its origin in the center.

So the two basic steps to convert from view coordinates and SpriteKit coordinates would be 1) inverting the y-axis so your origin is in the bottom left, and 2) ensuring that your SKScene frame matches your view frame.

I can test this out more fully in a bit and edit if there are any issues