How do I transform bounding boxes for view frustum culling?

Question

I have a scene graph with nodes holding transforms and bounding boxes in it,as well as a view frustum that I build each frame from the viewProjection matrix.However,the boxes have their 4 vertices' coordinates in the boxes' local space.What must I transform them by,to get them into the same space as the view frustum,so I can then check for intersection with the frustum?I tried bringing them into world space,but that was weird,since I have 50 world matrices(I use instancing and each instance has its own world/transform matrix)

Answer 1

Usually you want to collide BB and frustum in word space. Raw algo:

mtxViewProjTranspose = MtxTranspose(mtxView * mtxProj);
frustumViewWorld = frustumView * mtxViewProjTranspose;
foreach(i)
{
    bbWorld[i] = bb[i] * mtxWorld;
    bCull[i] = collide(BBWorld[i], frustumViewWorld );
}

Also you can do it in view space (but it not so good):

mtxProjTranspose = MtxTranspose(mtxProj);
frustumViewViewSpace = frustumView * mtxProjTranspose;
    foreach(i)
    {
        bbView[i] = bb[i] * mtxWorld[i] * mtxView;  // More calculations here
        bCull[i] = collide(bbView[i], frustumViewViewSpace );
    }

If you think that translating bounding boxes (4 verts) to world space is weird, think about how weird to do so with thousands of verts in meshes itself. =) As I understand, collision detection is far by more expensive, than this simple multiplication. But:

• To shorten list of collisions you can translate frustum to world space and roughly check which nodes are worth to collide with frustum and which are not. So you will also won't need to translate their BB. This depends on your scene implementation.
• Depending on your skills in HLSL, game style and targeting end-user hardware you can (or cannot) move BB transforms on GPU (Compute shader, and avaliable even on SM 4.0 with some limitations) as GPU usually do this with meshes VBs. But I don't think it gives you a bid gain. However it will be a good training to later move collision detection to GPU too.

Hope its help!