The problem at hand is part of a scientific simulation concerned with 2D growth within 3D space. The 2D shape grows by adding (triangular) segments to the previously grown shape.
Note that the actual segments in 3D have a thickness, thus, my code actually works with triangular prisms.
At one point, these 2D shapes (with whatever relative orientation and position) collide.
If one of the new triangular prisms intersect with previously inserted segments, I only want to insert the "part" of a segment which does not intersect with the previously inserted segments. As shown below for the segments labelled T1 and T2.
In the first step, I calculated all intersections edges to faces. I then used the CGAL Delaunay Triangulation package in 3D to mesh the resultant point set in a tetrahedral mesh. As a last step, I throw away all those tetrahedrons which intersect with previously inserted segments. This works beautifully in most cases - but I am convinced by now that the idea cannot work for fundamental reasons.
What's a more reliable way to compute this?
So you're looking to find the intersection of 2 convex hulls?
I think your correct that your approach won't work in all cases, but only in certain degenerate cases. For example if one convex hull is entirely inside the other then there are no face/edge intersections.
A more obviously robust approach is to start off with a convex hull big enough to enclose both your hulls (so basically representing all of space), and then for each plane in both your convex hulls partition this hull by that plane, throwing away the part on the "outside".
The result will be the intersection - which may be nothing.