I'm trying to calculate the view frustum corners in world space. I've implemented it by using the FOV and using the width/height of the planes and some vector math
However, a lot of examples simply state that you can multiply a NDC corner like (1,1,1) by the inverse viewProjection matrix. But when I do this I get somewhat different results. This is the code I'm using right now to test things:
float nearHeight = 2 * tan(mFOV / 2) * mNear;
float nearWidth = mNear * mRatio;
float farHeight = 2 * tan(mFOV / 2) * mFar;
float farWidth = mFar * mRatio;
glm::vec3 fc = mPos + mFront * mFar;
glm::vec3 nc = mPos + mFront * mNear;
mFrustum.frustumCorners[0] = fc + (mUp * farHeight / 2.0f) - (mRight * farWidth / 2.0f);
mFrustum.frustumCorners[1] = fc + (mUp * farHeight / 2.0f) + (mRight * farWidth / 2.0f);
mFrustum.frustumCorners[2] = fc - (mUp * farHeight / 2.0f) - (mRight * farWidth / 2.0f);
mFrustum.frustumCorners[3] = fc - (mUp * farHeight / 2.0f) + (mRight * farWidth / 2.0f);
mFrustum.frustumCorners[4] = nc + (mUp * nearHeight / 2.0f) - (mRight * nearWidth / 2.0f);
mFrustum.frustumCorners[5] = nc + (mUp * nearHeight / 2.0f) + (mRight * nearWidth / 2.0f);
mFrustum.frustumCorners[6] = nc - (mUp * nearHeight / 2.0f) - (mRight * nearWidth / 2.0f);
mFrustum.frustumCorners[7] = nc - (mUp * nearHeight / 2.0f) + (mRight * nearWidth / 2.0f);
glm::vec4 test(1.0f, 1.0f, 1.0f,1.0f);
glm::vec4 test2(-1.0f, -1.0f, -1.0f, 1.0f);
glm::mat4 testingMatrix = glm::inverse(mProjectionMatrix * getViewMatrix());
test = testingMatrix*test;
test2 = testingMatrix*test2;
test2.x /= test2.w;
test2.y /= test2.w;
test2.z /= test2.w;
test.x /= test.w;
test.y /= test.w;
test.z /= test.w;
Now both of these results give an accurate z value for [near,far] = [1, 10000] but the x values are off by quite a bit while the y values are pretty much the same. I was just wonder which way is the most accurate one?
Inverse viewProjection
Regular calculation