I am having trouble retrieving a correct result from a boolean operation. I am creating a (supposedly) solid prism by sweeping a face, and attempting to boolean cut that prism into a box primitive. I make the prism as follows:
1) Create a Geom_BSplineSurface, which is a 5x5 grid of (coplanar) control points evenly spread out spatially from (0,0,0) to (4,4,0).
2) Create a face from the Geom-BSplineSurface with BRepBuilderAPI_MakeFace(surface, true, 0.0001).
3) Create a prism from the face with BRepPrimAPI_MakePrism(face, gp_Vec(0,0,3), true). Since the face is effectively just a square, this generates a box-shaped solid.
4) I then modify the transform of the prism shape so it partially overlaps a box primitive (generated from BRepPrimAPI_MakeBox), and perform a boolean cut operation with BRepAlgoAPI_Cut to remove the prism from the box.
Finally, I use BRepMesh_IncrementalMesh to create a mesh representation of the result to display with my renderer.
The problem is that the boolean result mesh always appears the same, regardless of whether the prism is cutting into it. I should note that doing the exact same procedure but cutting the box with another plain box primitive produces the expected result (a box with a portion cut out where the cutting box is located), so it seems to be some issue with the prism.
What's more, if I inspect the actual topology of the boolean result, I can see it has changed from the original target box, and appears to have added edges related to where the cutting prism overlaps, so at least I know something is happening. I also see it now has 2 vertices with an "internal" orientation as well. But the positioning of the vertices/triangles from the tessellated mesh still appears to be all on the external surface of the original box (so visually it looks the same, but you can see new vertices where the prism is intersecting the surface).
It is unclear to me whether the boolean operation itself is not performing as expected, possibly due to something about the prism (I notice the prism is a Solid with forward orientation, but contains a Shell with Reversed orientation? I tried reversing the Shell orientation, but this did not help), or whether it is just a matter of how the mesh is being retrieved.
I retrieve the mesh from BRepMesh_IncrementalMesh by using TopExp_Explorer to iterate over the faces in the boolean result, and collecting the node positions and triangle indices.
Any help/ideas is greatly appreciated as the lack of a proper result is currently blocking progress in my work.