DEVHIDE
Login Or Sign up

Motion Blur with SCNTechnique

1.6k Views Asked by At

I've been trying to get motion blur working with SCNTechnique for a few days now and I'm nowhere close to what I want. I've asked a similar question on the Apple forums but they're dead. So I'd thought I'd write a more exhaustive description of what I'm trying to achieve with code.

The setup

I have a character on screen with some enemies. The character is a square, the enemies are circles - simplified for this example.

I am using SceneKit with Metal. The camera is fixed.

The Goal

When the circles/enemies move I want them to have motion blur. When the character moves he should not.

Proposed Idea

Write a multipass SCNTechnique that can handle this, I'm assuming that this is a way of doing what I want.

Pass one: Render just the enemies/circles to another buffer

Pass two: Apply motion blur to this buffer

Pass three: Render the motion blur buffer with the original scene.

Side note: Pass two will be tricky as I imagine each object has it's own direction and blur would need to follow suit.

So, here's how I setup the objects in SceneKit

class Character : SCNNode
{
    override init() {
        super.init()

        let img = UIImage(named: "texture1")!

        material = SCNMaterial()
        material.diffuse.contents = img
        material.ambient.contents = img

        let geometry = SCNSphere(radius: 40)
        geometry.materials = [material]

        self.categoryBitMask = 1
    }
}

class Enemy : SCNNode
{
    override init() {
        super.init()

        let img = UIImage(named: "texture2")!

        material = SCNMaterial()
        material.diffuse.contents = img
        material.ambient.contents = img

        let geometry = SCNSphere(radius: 40)
        geometry.materials = [material]

        self.categoryBitMask = 2
    }
}

I can add these to the scene and they look fine. I can move them both with SCNActions and they move correctly.

Now for how I'm attempting motion blur

First Pass

I want to extract just enemies for this pass so this part of the technique looks like the following, note the category mask draws just the enemies for the scene.

It outputs this to my custom "enemiesColor" target buffer. Note: DRAW_SCENE

<key>drawEnemies</key>
    <dict>
        <key>draw</key>
        <string>DRAW_SCENE</string>
        <key>includeCategoryMask</key>
        <integer>2</integer>
        <key>excludeCategoryMask</key>
        <integer>1</integer>
        <key>program</key>
        <string>doesntexist</string>
        <key>metalVertexShader</key>
        <string>multi_vertex</string>
        <key>metalFragmentShader</key>
        <string>multi_fragment_vert</string>
        <key>inputs</key>
        <dict>
            <key>colorSampler</key>
            <string>COLOR</string>
            <key>a_texcoord</key>
            <string>a_texcoord-symbol</string>
            <key>aPos</key>
            <string>vertexSymbol</string>
        </dict>
        <key>outputs</key>
        <dict>
            <key>color</key>
            <string>enemiesColor</string>
        </dict>
    </dict>

The metal shader for this is:

#include <metal_stdlib>
using namespace metal;
#include <SceneKit/scn_metal>

struct custom_node_t3 {
    float4x4 modelTransform;
    float4x4 modelViewTransform;
    float4x4 normalTransform;
    float4x4 modelViewProjectionTransform;
};

struct custom_vertex_t
{
    float4 position [[attribute(SCNVertexSemanticPosition)]];
    float2 a_texcoord [[ attribute(SCNVertexSemanticTexcoord0) ]];
    //SCNGeometrySourceSemanticTexcoord
};


constexpr sampler s = sampler(coord::normalized,
                          address::repeat,
                          filter::linear);

struct out_vertex_t
{
    float4 position [[position]];
    float2 texcoord;
};

vertex out_vertex_t multi_vertex(custom_vertex_t in [[stage_in]],
                                    constant custom_node_t3& scn_node [[buffer(0)]])
{
    out_vertex_t out;
    out.texcoord = in.a_texcoord;
    out.position = scn_node.modelViewProjectionTransform *     float4(in.position.xyz, 1.0);

    return out;
};



fragment half4 multi_fragment_vert(out_vertex_t vert [[stage_in]],
                               constant SCNSceneBuffer& scn_frame     [[buffer(0)]],
                                      texture2d<float, access::sample>     colorSampler [[texture(0)]])
{

    float4 FragmentColor = colorSampler.sample( s, vert.texcoord);
    return half4(FragmentColor);
};

Second Pass

I want to blur the "enemiesColor" buffer, at the moment it's very rough, so I'm just using a gaussian hack at the moment.

I take in the "enemiesColor" buffer as an input and blur it, I output this as a new buffer: "enemyColor" Note: DRAW_QUAD

The technique for this pass looks like:

<key>blurEnemies</key>
    <dict>
        <key>draw</key>
        <string>DRAW_QUAD</string>
        <key>program</key>
        <string>doesntexist</string>
        <key>metalVertexShader</key>
        <string>blur_vertex</string>
        <key>metalFragmentShader</key>
        <string>blur_fragment_vert</string>
        <key>inputs</key>
        <dict>
            <key>colorSampler</key>
            <string>COLOR</string>
            <key>enemyColor</key>
            <string>enemiesColor</string>
            <key>a_texcoord</key>
            <string>a_texcoord-symbol</string>
        </dict>
        <key>outputs</key>
        <dict>
            <key>color</key>
            <string>chrisColor</string>
        </dict>
    </dict>

and the shader is:

// http://rastergrid.com/blog/2010/09/efficient-gaussian-blur-with-linear-sampling/
constant float offset[] = { 0.0, 1.0, 2.0, 3.0, 4.0 };
constant float weight[] = { 0.2270270270, 0.1945945946, 0.1216216216, 0.0540540541, 0.0162162162 };


vertex out_vertex_t blur_vertex(custom_vertex_t in [[stage_in]],
                             constant custom_node_t3& scn_node [[buffer(0)]])
{
    out_vertex_t out;
    out.position = in.position;
    out.texcoord = float2((in.position.x + 1.0) * 0.5 , (in.position.y + 1.0) * -0.5);

    return out;
};

fragment half4 blur_fragment_vert(out_vertex_t vert [[stage_in]],
                               texture2d<float, access::sample> colorSampler [[texture(0)]],
                                   texture2d<float, access::sample> enemyColor [[texture(1)]])
{

    float4 enemySample = enemyColor.sample(s, vert.texcoord);

    if (enemySample.a == 0)
    {
        //gl_LastFragData[0]
        return half4(0.0 ,1.0 ,0.0, 0.5);
    }


    float4 FragmentColor = colorSampler.sample( s, vert.texcoord) * weight[0];
    for (int i=1; i<5; i++) {
        FragmentColor += colorSampler.sample( s, ( vert.texcoord + float2(0.0, offset[i])/224.0 ) ) * weight[i];
        FragmentColor += colorSampler.sample( s, ( vert.texcoord - float2(0.0, offset[i])/224.0 ) ) * weight[i];
    }
    return half4(FragmentColor);


};

Third pass

Where things get even messier, I want to then apply the blurred "enemyColor" buffer with the original scene.

My first thought was, well why can't I just overlay the result. I looked into blend modes but found no luck.

Then I thought, maybe I can just re-render the scene and add the "enemyColor" and the new "color" buffer together (there's probably optimisations somewhere if this even remotely works)

Note: DRAW_SCENE

So the technique is:

<key>blendTogether</key>
    <dict>
        <key>draw</key>
        <string>DRAW_SCENE</string>
        <key>program</key>
        <string>doesntexist</string>
        <key>metalVertexShader</key>
        <string>plain_vertex</string>
        <key>metalFragmentShader</key>
        <string>plain_fragment_vert</string>
        <key>inputs</key>
        <dict>
            <key>colorSampler</key>
            <string>COLOR</string>
            <key>aPos</key>
            <string>vertexSymbol</string>
            <key>a_texcoord</key>
            <string>a_texcoord-symbol</string>
        </dict>
        <key>outputs</key>
        <dict>
            <key>color</key>
            <string>COLOR</string>
        </dict>
    </dict>

and the shader:

    vertex out_vertex_t plain_vertex(custom_vertex_t in [[stage_in]],
                             constant SCNSceneBuffer& scn_frame [[buffer(0)]],
                            constant custom_node_t3& scn_node [[buffer(1)]])
{
    out_vertex_t out;

    out.position = scn_node.modelViewProjectionTransform * float4(in.position.xyz, 1.0);
    out.texcoord = in.a_texcoord;

    return out;
};

fragment half4 plain_fragment_vert(out_vertex_t vert [[stage_in]],
                               texture2d<float, access::sample> colorSampler [[texture(0)]])
{

    float4 FragmentColor = colorSampler.sample( s, vert.texcoord);
    return half4(FragmentColor);

};

At the end of this, my scene renders, but I'm just not getting the desired effects.

The first question would be...is motion blur possible with a system like this, I don't want to chase this any-longer if its not.

The second is, where am I going wrong?

The complete technique for completion:

    <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>passes</key>
    <dict>
        <key>drawEnemies</key>
        <dict>
            <key>draw</key>
            <string>DRAW_SCENE</string>
            <key>includeCategoryMask</key>
            <integer>2</integer>
            <key>excludeCategoryMask</key>
            <integer>1</integer>
            <key>program</key>
            <string>doesntexist</string>
            <key>metalVertexShader</key>
            <string>multi_vertex</string>
            <key>metalFragmentShader</key>
            <string>multi_fragment_vert</string>
            <key>inputs</key>
            <dict>
                <key>colorSampler</key>
                <string>COLOR</string>
                <key>a_texcoord</key>
                <string>a_texcoord-symbol</string>
                <key>aPos</key>
                <string>vertexSymbol</string>
            </dict>
            <key>outputs</key>
            <dict>
                <key>color</key>
                <string>enemiesColor</string>
            </dict>
        </dict>
        <key>blurEnemies</key>
        <dict>
            <key>draw</key>
            <string>DRAW_QUAD</string>
            <key>program</key>
            <string>doesntexist</string>
            <key>metalVertexShader</key>
            <string>blur_vertex</string>
            <key>metalFragmentShader</key>
            <string>blur_fragment_vert</string>
            <key>inputs</key>
            <dict>
                <key>colorSampler</key>
                <string>COLOR</string>
                <key>enemyColor</key>
                <string>enemiesColor</string>
                <key>a_texcoord</key>
                <string>a_texcoord-symbol</string>
            </dict>
            <key>outputs</key>
            <dict>
                <key>color</key>
                <string>chrisColor</string>
            </dict>
        </dict>
        <key>blendTogether</key>
        <dict>
            <key>draw</key>
            <string>DRAW_SCENE</string>
            <key>program</key>
            <string>doesntexist</string>
            <key>metalVertexShader</key>
            <string>plain_vertex</string>
            <key>metalFragmentShader</key>
            <string>plain_fragment_vert</string>
            <key>inputs</key>
            <dict>
                <key>colorSampler</key>
                <string>COLOR</string>
                <key>aPos</key>
                <string>vertexSymbol</string>
                <key>a_texcoord</key>
                <string>a_texcoord-symbol</string>
            </dict>
            <key>outputs</key>
            <dict>
                <key>color</key>
                <string>COLOR</string>
            </dict>
        </dict>
    </dict>
    <key>sequence</key>
    <array>
        <string>blendTogether</string>
    </array>
    <key>targets</key>
    <dict>
        <key>enemiesColor</key>
        <dict>
            <key>type</key>
            <string>color</string>
        </dict>
        <key>chrisColor</key>
        <dict>
            <key>type</key>
            <string>color</string>
        </dict>
    </dict>
    <key>symbols</key>
    <dict>
        <key>a_texcoord-symbol</key>
        <dict>
            <key>semantic</key>
            <string>texcoord</string>
        </dict>
        <key>vertexSymbol</key>
        <dict>
            <key>semantic</key>
            <string>vertex</string>
        </dict>
    </dict>
</dict>
</plist>
shader scenekit metal scntechnique
Original Q&A
2

There are 2 best solutions below

3
On

If you can wait until macOS Sierra/iOS 10, the new SceneKit HDI camera supports motion blur.

Video: https://developer.apple.com/videos/play/wwdc2016/609/

Source; https://developer.apple.com/library/prerelease/content/samplecode/Badger/Introduction/Intro.html

0
On

This answer assumes that your objects are 2D objects (as implied by the 'circles' and 'squares' in your description) embedded in SceneKit's 3D world.

I think multi-pass rendering with SCNTechnique is probably not the right solution for what you are trying to achieve.

This is how I would approach it:

Add a (fairly wide) transparent margin to your texture for the circle objects and make the SceneKit circle objects larger so that the opaque part has the correct size. The width of the transparent margin is the maximal length for the motion blur trails the objects can have. So if the circle enemies can move quite far in one frame, you will want a wide margin.

Use a custom SCNProgram to specify a fragment shader for your circle objects. This is where you would implement the motion blur rendering. You will need to pass the object velocity into the shader as a custom variable (see the "Custom Variables" section of the SCNProgram documentation). Also, you will need to convert/transform the velocity vector into the 2D texture coordinate system of the circle object.

In the fragment shader you then sample the texture along the velocity vector and average the sampled colors. You might want to choose the number of samples depending on the magnitude of the velocity: The faster the object moves, the more samples you will want to use. A fixed number of samples might be fine too though, as long as it is high enough.

Illustration of fragment shader for motion blur

In the illustration above the small green square shows an example pixel for which the fragment shader is being evaluated. And the 4 yellow dots show the sample positions at which you might evaluate the texture. In this case 2 samples hit the transparent margin in the texture and the other 2 fall into the opaque part. So the output color would have an alpha of 0.5 in this case.

You can also experiment with the sample weights and maybe use a higher weight for the current position (the sample that is inside the pixel) depending on the look you are after.

Related Questions in SHADER

Related Questions in SCENEKIT

Related Questions in METAL

Related Questions in SCNTECHNIQUE

Trending Questions

Popular # Hahtags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Popular Questions

Copyright © 2021 Jogjafile Inc.