Using a MTLTexture as the environment map of a SCNScene

Question

I want to set a MTLTexture object as the environment map of a scene, as it seems to be possible according to the documentation. I can set the environment map to be a UIImage with the following code:

let roomImage = UIImage(named: "room")
scene.lightingEnvironment.contents = roomImage

This works and I see the reflection of the image on my metallic objects. I tried converting the image to a MTLTexture and setting it as the environment map with the following code:

let roomImage = UIImage(named: "room")
let loader = MTKTextureLoader(device: MTLCreateSystemDefaultDevice()!)
let envMap = try? loader.newTexture(cgImage: (roomImage?.cgImage)!, options: nil)
scene.lightingEnvironment.contents = envMap

However this does not work and I end up with a blank environment map with no reflection on my objects.

Also, instead of setting the options as nil, I tried setting the MTKTextureLoader.Option.textureUsage key with every possible value it can get, but that didn't work either.

Edit: You can have a look at the example project in this repo and use it to reproduce this use case.

Answer 1

Lighting SCN Environment with an MTK texture

Using Xcode 13.3.1 on macOS 12.3.1 for iOS 15.4 app.

The trick is, the environment lighting requires a cube texture, not a flat image.

• Create 6 square images for MetalKit cube texture

• in Xcode Assets folder create Cube Texture Set

• place textures to their corresponding slots

• mirror images horizontally and vertically, if needed

Paste the code:

import ARKit
import MetalKit

class ViewController: UIViewController {

    @IBOutlet var sceneView: ARSCNView!
    
    override func viewDidLoad() {
        super.viewDidLoad()    
        let scene = SCNScene()
        
        let imageName = "CubeTextureSet"
        let textureLoader = MTKTextureLoader(device: sceneView.device!)

        let environmentMap = try! textureLoader.newTexture(name: imageName, 
                                                    scaleFactor: 2, 
                                                         bundle: .main, 
                                                        options: nil)
        
        let daeScene = SCNScene(named: "art.scnassets/testCube.dae")!

        let model = daeScene.rootNode.childNode(withName: "polyCube", 
                                             recursively: true)!
        
        scene.lightingEnvironment.contents = environmentMap
        scene.lightingEnvironment.intensity = 2.5
        scene.background.contents = environmentMap

        sceneView.scene = scene
        sceneView.allowsCameraControl = true
        scene.rootNode.addChildNode(model)
    }
}

Apply metallic materials to models. Now MTL environment lighting is On.

If you need a procedural skybox texture – use MDLSkyCubeTexture class.

Also, this post may be useful for you.