How to make a GUI to visually add Mattertags into a Matterport scene?

Question

There are 2 examples in the Matterport SDK for Embeds documentation to show how to place Mattertags in a scene:

• The Intersection Inspector which only allows you to see coordinates for placing a Mattertag where the cursor is if you wait a little bit ... Not very user friendly, you need to copy the coordinates manually in your program.
• The Transient Tags Editor which enable you to interactively place multiple Mattertags visually, edit them and then to extract them easily in a JSON file ...

I was wondering how to reproduce the Transient Tags Editor visual UX as I would like to use it in an application.

Answer 1

Insert Mattertags into the model visually

The source code of the app of the Transient Tags Editor is privately hosted on github (Maybe because it doesn't run perfectly on Firefox?), unlike the source code of the Intersection Inspector which is publicly hosted on JSFiddle.

But the user friendliness of the Transient Tags Editor intrigued me and I wanted to understand the difference between the two tools Matterport SDK provides to find out Mattertags coordinates.

How the Intersection Inspector works

The Intersection Inspector uses a timer to display a button at the position of the Pointer when the user does not move the pointer for more than one second. The user can then click the button to see the Mattertag coordinates and copy them manually ...

To achieve that, it needs the current Camera position, which it obtains by observing the camera's pose property:

    var poseCache;
    mpSdk.Camera.pose.subscribe(function(pose) {
        poseCache = pose;
    });

Also, it needs the current Pointer position, which it obtains by observing the pointer's intersection property:

    var intersectionCache;
    mpSdk.Pointer.intersection.subscribe(function(intersection) {
        intersectionCache = intersection;
        intersectionCache.time = new Date().getTime();
        button.style.display = 'none';
        buttonDisplayed = false;
    });

※ An intersection event is triggered the user moves the pointer, so we hide the button to make sure it is not displayed before the one second delay is over.

Then, a timer is set up using setInterval() to display the button at the right time:

    setInterval(() => {
        // ...
    }, 16);

In the timer callback, we check wether all the conditions to display the button are met ...

• First, check we have the information we need:

    setInterval(() => {
        if (!intersectionCache || !poseCache) {
            return;
        }
        // ...
    }, 16);

• Then, check one second has elapsed since the last intersection event was received, or we wait the next tick to check again:

    var delayBeforeShow = 1000;
    setInterval(() => {
        if (!intersectionCache || !poseCache) {
            return;
        }
        const nextShow = intersectionCache.time + delayBeforeShow;
        if (new Date().getTime() > nextShow) {
            // ...
        }
    }, 16);

• Finally, we check the button is not already being displayed:

    var delayBeforeShow = 1000;
    var buttonDisplayed = false;
    setInterval(() => {
        if (!intersectionCache || !poseCache) {
            return;
        }
        const nextShow = intersectionCache.time + delayBeforeShow;
        if (new Date().getTime() > nextShow) {
            if (buttonDisplayed) {
                return;
            }
            // ...
        }
    }, 16);

Once the conditions are met, we can display the button using Conversion.worldToScreen() to get the screen coordinate of the pointer :

    // ...
    setInterval(() => {
        // ...
        if (new Date().getTime() > nextShow) {
            // ...
            var size = {
                w: iframe.clientWidth,
                h: iframe.clientHeight,
            };
            var coord = mpSdk.Conversion.worldToScreen(intersectionCache.position, poseCache, size);
            button.style.left = `${coord.x - 25}px`;
            button.style.top = `${coord.y - 22}px`;
            button.style.display = 'block';
            buttonDisplayed = true;
        }
    }, 16);

The button is a simple HTML button hidden by default using display: none; and positioned relative to the body with position: absolute;.

When the user clicks the button, the current coordinates of the pointer are displayed in a <div> tag above the <iframe> and the button is hidden:

    button.addEventListener('click', function() {
        text.innerHTML = `position: ${pointToString(intersectionCache.position)}\nnormal: ${pointToString(intersectionCache.normal)}\nfloorId: ${intersectionCache.floorId}`;
        button.style.display = 'none';
        iframe.focus();
    });

The coordinates are formatted using the following function:

function pointToString(point) {
  var x = point.x.toFixed(3);
  var y = point.y.toFixed(3);
  var z = point.z.toFixed(3);

  return `{ x: ${x}, y: ${y}, z: ${z} }`;
}

Now, let's see how the easier-to-use and user-friendlier Transient Tags Editor interface works ...

How the Transient Tag Editor works

The Intersection Inspector is enough if you just have a few __Mattertag__s to set permanently in a few models in your application. But if you need your users to set tags interactively in models, something like the Transient Tags Editor's GUI is a better starting point.

The main advantage of using the Transient Tags Editor is that you can see how the Mattertag will be displayed before creating it and! That allows you to place the tag precisely without trial and error ...

To add a tag, you must click on the "Place New Tag" button to enter the "add tag" mode, then you can place one new tag anywhere you want.

We will only focus on that aspect of the editor and produce a simplified code sample that only add tags:

• As the user move a tag along the pointer when in "add tag" mode, the first step is to create a new tag and place it. Let's create a function for that using Mattertag.add():

    function addTag() {
        if(!tag){
            mpSdk.Mattertag.add([{
                label: "Matterport Tag",
                description: "",
                anchorPosition: {x: 0, y: 0, z: 0},
                stemVector: {x:0, y: 0, z: 0},
                color: {r: 1, g: 0, b: 0},
            }])
            .then((sid) => {
                tag = sid[0];
            })
            .catch( (e) => {
                console.error(e);
            })
        }
    }

• Then we will have to place the tag at a position near the pointer, and update its position as the user moves the pointer, so let's create a function for that using Mattertag.editPosition():

    function updateTagPos(newPos, newNorm=undefined, scale=undefined){
        if(!newPos) return;
        if(!scale) scale = .33;
        if(!newNorm) newNorm = {x: 0, y: 1, z: 0};

        mpSdk.Mattertag.editPosition(tag, {
            anchorPosition: newPos,
            stemVector: {
                x: scale * newNorm.x,
                y: scale * newNorm.y,
                z: scale * newNorm.z,
            }
        })
        .catch(e =>{
            console.error(e);
            tag = null;
        });
    }

As you can see the updateTagPos() function takes 3 parameters:

1. newPos: the new anchor position for the Mattertag.
2. newNorm: an optional new stem vector for the Mattertag.
3. scale: an optional new scale for the stem of the Mattertag.

• To update the tag position as the user moves the pointer, let's observe the pointer's intersection property to call updateTagPos():

    mpSdk.Pointer.intersection.subscribe(intersectionData => {
        if(tag){
            if(intersectionData.object === 'intersectedobject.model' || intersectionData.object === 'intersectedobject.sweep'){
                updateTagPos(intersectionData.position, intersectionData.normal);
            }
        }
    });

• To place the new tag, the user simply clicks their mouse button, the Transient Tags Editor provides its own version of the document.activeElement method for intercepting clicks on the <iframe> (but does not work with Firefox so the editor use a quite complex workaround):

    function focusDetect(){
        const eventListener = window.addEventListener('blur', function() {
            if (document.activeElement === iframe) {
                placeTag(); //function you want to call on click
                setTimeout(function(){ window.focus(); }, 0);
            }
            window.removeEventListener('blur', eventListener );
        });
    }

But, we will use our version which works better with Firefox (But still stop working after the first click in Firefox for whatever reason):

    window.addEventListener('blur',function(){  
        window.setTimeout(function () { 
            if (document.activeElement === iframe) {
                placeTag(); //function you want to call on click
                window.focus()
                addTag();
            }
        }, 0);
    });

• Finally, let's the function that navigates to the new tag and opens its billboard, usingMattertag.navigateToTag()

    function placeTag(){
        if(tag) mpSdk.Mattertag.navigateToTag(tag, mpSdk.Mattertag.Transition.INSTANT);
        tag = null;
    }

Simple Editor Code Sample

First, the complete JavaScript source code:

"use strict";

const sdkKey = "aaaaXaaaXaaaXaaaXaaaXaaa"
const modelSid = "iL4RdJqi2yK";

let iframe;
let tag;

document.addEventListener("DOMContentLoaded", () => {
    iframe = document.querySelector('.showcase');
    iframe.setAttribute('src', `https://my.matterport.com/show/?m=${modelSid}&help=0&play=1&qs=1&gt=0&hr=0`);
    iframe.addEventListener('load', () => showcaseLoader(iframe));
});

function showcaseLoader(iframe){
    try{
        window.MP_SDK.connect(iframe, sdkKey, '3.10')
        .then(loadedShowcaseHandler)
        .catch(console.error);
    } catch(e){
        console.error(e);
    }
}

function loadedShowcaseHandler(mpSdk){
    addTag()

    function placeTag(){
        if(tag) mpSdk.Mattertag.navigateToTag(tag, mpSdk.Mattertag.Transition.INSTANT);
        tag = null;
    }

    window.addEventListener('blur',function(){  
        window.setTimeout(function () { 
            if (document.activeElement === iframe) {
                placeTag(); //function you want to call on click
                window.focus()
                addTag();
            }
        }, 0);
    });
    function updateTagPos(newPos, newNorm=undefined, scale=undefined){
        if(!newPos) return;
        if(!scale) scale = .33;
        if(!newNorm) newNorm = {x: 0, y: 1, z: 0};

        mpSdk.Mattertag.editPosition(tag, {
            anchorPosition: newPos,
            stemVector: {
                x: scale * newNorm.x,
                y: scale * newNorm.y,
                z: scale * newNorm.z,
            }
        })
        .catch(e =>{
            console.error(e);
            tag = null;
        });
    }

    mpSdk.Pointer.intersection.subscribe(intersectionData => {
        if(tag){
            if(intersectionData.object === 'intersectedobject.model' || intersectionData.object === 'intersectedobject.sweep'){
                updateTagPos(intersectionData.position, intersectionData.normal);
            }
        }
    });

    function addTag() {
        if(!tag){
            mpSdk.Mattertag.add([{
                label: "Matterport Tag",
                description: "",
                anchorPosition: {x: 0, y: 0, z: 0},
                stemVector: {x:0, y: 0, z: 0},
                color: {r: 1, g: 0, b: 0},
            }])
            .then((sid) => {
                tag = sid[0];
            })
            .catch( (e) => {
                console.error(e);
            })
        }
    }
} // loadedShowcaseHandler

The HTML source code:

<!DOCTYPE html>
<html>
    <head>
        <title>Transient Tag Editor</title>
        <style>
            #showcase {
                width: 100%;
                height: 100vh;
            }
        </style>
        <script src="https://static.matterport.com/showcase-sdk/2.0.1-0-g64e7e88/sdk.js"></script>
        <script src="/js/app-editor.js" type="text/javascript" defer></script>
    </head>
    <body>
        <iframe id="showcase" frameborder="0"  allowFullScreen allow="xr-spatial-tracking"></iframe>
    </body>
</html>

It works!

Complete Code

The complete code for this sample and others is available on github: github.com/loic-meister-guild/pj-matterport-sdk-2021-tutorial

See Also

• Matterport SDK 2021 Tutorial
• Node.js + Express Tutorial for 2021
• How to detect a click event on a cross domain iframe