OpenGL Object got Distored after dragging in the Screen Space

Question

I wanted to drag a object with the mouse position offset in OpenGL. And the rendering pipeline goes like this.

gl_Position = projection * view * model * vec4(aPos, 1.0);

So I thought I should inverse (projection * view) then multiply them agian to get that transformation which will apply to the model matrix.

glm::mat4 CalculateDragTransform(double xOffset, double yOffset) {
    double x = (2 * xOffset) / m_width;
    double y = (2 * yOffset) / m_height;
    glm::mat4 projection = glm::perspective(glm::radians(m_camera->GetFOV()),
                                            (float) m_width /
                                            (float) m_height, 0.1f,
                                            100.0f);
    glm::mat4 view = m_camera->GetViewMatrix();
    glm::mat4 transform = glm::translate(glm::mat4(1.0), glm::vec3(x, y, 0));
    return glm::inverse(projection * view) * transform * projection * view;

m_objects[index]->SetModelMatrix(transform * m_objects[index]->GetModelMatrix());

The object was following the mouse movement when i dragged it. But the object got distorted massively if I view it from other angle. So What did I do wrong? Should (projection * view) be inverted?

Dice Before Dragging:

Dice After Dragging:

You can see the distortion from other angles:

A more obivous example of the distortion:

Update: I managed to fix the issue (kind of) by only keeping the position translation. So I suspect there's something wrong with the upper left 3x3 matrix in the transformation. When I debug it, I found out that the first three column of the forth row are not zeros, which is absolutely incorrect! But I couldn't understand what caused it.. This phenomena only happens when I use glm::perspective as projection matrix. Using glm::ortho is also a solution without having to discard the upper-left 3x3 matrix.

    glm::mat4 view = m_camera->GetViewMatrix();
    glm::mat4 transform = glm::translate(glm::mat4(1.0), glm::vec3(x, y, 0));
    transform = glm::inverse(projection * view) * transform * projection * view;
    float x = transform[3][0];
    float y = transform[3][1];
    float z = transform[3][2];
    transform = glm::translate(glm::mat4(1.0f), glm::vec3(x, y, z));

Answer 1

You could omit the matrix transformations and just query the cursor position on your screen. If you are using GLFW, it would look like this

glm::vec2 mouse_pos = glm::vec2(0.f);
void mouse_callback(GLFWwindow* window, double xpos, double ypos) {
    mouse_pos = glm::vec2(xpos / width - 0.5, 1.0 - ypos / height - 0.5);
}

and you would use it with

GLFWwindow* window = ... ;
glfwSetCursorPosCallback(window, mouse_callback);

This way, we can get our mouse position on screen, where the center is glm::vec2(0) and both x and y range from +1.0 to -1.0. For more information on how GLRF handles the mouse position see https://www.glfw.org/docs/3.0/group__input.html#ga01d37b6c40133676b9cea60ca1d7c0cc.

Let's define the camera with the camera coordinate system, where the vector u points to the right side of the screen, the vector v points to the top of the screen and the vector w points in the negative viewing direction.

To get the 3d position, the camera view direction a.k.a. -w can be rotated with the field of view (fov) angle.

float inv_aspect_ratio = height / width;
glm::mat4 rotation_horizontal = glm::rotate(glm::mat4(1.f), -0.5f * fov * mouse_pos.x, v);
glm::mat4 rotation_vertical = glm::rotate(glm::mat4(1.f), -0.5f * fov * inv_aspect_ratio
    * mouse_pos.y, u);

where the fov was defined for the width. This way, the fov for the height can be calculated by multiplying the inverse aspect ratio of the screen.

For the final step, the rotation is applied to the negative camera direction -w and a distance from the camera origin is specified.

glm::vec3 dice_direction = glm::vec3(rotation_vertical * rotation_horizontal * glm::vec4(-w, 0.f));
float dice_distance_from_camera
glm::vec3 dice_position = dice_direction * dice_distance_from_camera;