add mouseevents to webgl objects

Question

im using xtk to visualize medical data in a webgl canvas. currently im playing around with this lesson:

lesson 10

this library is pretty good but not very well documented. i want to get rid of that gui and add some mouseevents. if i load the mesh from the gui how can i add a mouse event to the mesh? i actually don't know where to start. it's a little bit confusing to get started with this library....

i tried

mesh.click(function(){
    alert("yes");
  })

or

mesh.mousedown(function(){
    alert("yes");
}

Answer 1

DOM events are not supported but you can do it with xtk. Check out this JSFiddle

http://jsfiddle.net/haehn/r7Ugf/

// create and initialize a 3D renderer
var r = new X.renderer3D();
r.init();

// create a cube and a sphere
cube = new X.cube();
sphere = new X.sphere();
sphere.center = [-20, 0, 0];

r.interactor.onMouseMove = function() {

    // grab the current mouse position
    var _pos = r.interactor.mousePosition;

    // pick the current object
    var _id = r.pick(_pos[0], _pos[1]);

    if (_id != 0) {

        // grab the object and turn it red
        r.get(_id).color = [1, 0, 0];

    } else {

        // no object under the mouse
        cube.color = [1, 1, 1];
        sphere.color = [1, 1, 1];

    }

    r.render();

}

r.interactor.onMouseDown = function(left, middle, right) {

    // only observe right mouse clicks        
    if (!right) return;

    // grab the current mouse position
    var _pos = r.interactor.mousePosition;

    // pick the current object
    var _id = r.pick(_pos[0], _pos[1]);

    if (_id == sphere.id) {

        // turn the sphere green
        sphere.color = [0, 1, 0];
        r.render();

    }

}

r.add(cube); // add the cube to the renderer
r.add(sphere); // and the sphere as well
r.render(); // ..and render it

Easy, no?

Answer 2

Objects rendered in WebGL are not part of the DOM, and as such don't generate events like DOM elements do. This means that for events like these you have to implement the mouse interaction code yourself.

Traditionally in WebGL/OpenGL this process is known as "Picking", and there's several decent resources for it online. (For example: http://webgldemos.thoughtsincomputation.com/engine_tests/picking) The core process is something like this, though:

• For each pickable object in your scene, assign it a color. Put this in a lookup table somewhere
• Re-render the entire scene to a texture, rendering each pickable object with it's assigned color
• Once the scene is rendered, determine your mouse coordinates and read back the color of the texture at that X/Y.
• Fetch the object associated with that color from your lookup table. This is the object your mouse cursor is pointing at!

As you can see, while not a difficult method conceptually this also involves several mid-level WebGL topics, such as rendering to a texture, and as such is not usually recommended for beginners. I'm not sure if there are any features in xtk to assist with this (honestly I had never heard of the library before your post), but I would guess that this is something that you'll have to implement on your own.

Answer 3

XTK implements picking the way Toji explained (i.e. with a frameBuffer where every object is rendered in a different RGBA "color"). It will work while you have less than 255^4 objects, so almost always. There are other methods like unprojecting but they would be longer I think.

So with X.renderer.pick and X.renderer.get you can find the object under the mouse and change its properties. However for the moment you can only change vizualisation properties (see the setGetter and setSetter in every class) but you cannot move an X.object (since X.object._transform attribute is private and there is no getter/setter for it yet).

That's something interesting to deal with : adding a pair of getter/setter for X.object's transform would allow, for example, an user to put medical stuff (modelized by a mesh or something else) in the scene and place to mesure distances or see if it will fit for an operation or something like that. Shouldn't be a good idea Haehn ? And it's a minor change in the framework.