How to compute a bounding box of a TopoDS_Shape in a specific coordinate system (not global)

If you want computing AABB (axes-aligned bounding box) of shape within a local coordinate system, the only thing to do is to calculate a proper transformation to this local system and apply it to shape or AABB.

Lets start from defining an auxiliary tool for printing AABB (Bnd_Box):

  auto printBox = [](const Bnd_Box& theBox, const char* theName) {
    gp_Pnt aMin = theBox.CornerMin(), aMax = theBox.CornerMax();
    gp_XYZ aDim = aMax.XYZ() - aMin.XYZ();
    std::cout << theName << "\n"
      << "  MIN: " << aMin.X() << " " << aMin.Y() << " " << aMin.Z() << "\n"
      << "  MAX: " << aMax.X() << " " << aMax.Y() << " " << aMax.Z() << "\n"
      << "  DIM: " << aDim.X() << " " << aDim.Y() << " " << aDim.Z() << "\n";
  };

and construct a trivially-oriented Cone:

  TopoDS_Shape aCone0 = BRepPrim_Cone(gp::XOY(), 100, 0, 400).Shell();
  gp_Trsf aTrans; // add some location for testing
  aTrans.SetTranslation(gp::Origin(), gp_Pnt(50, 0, 0));
  aCone0.Location(aTrans);
  DBRep::Set("c0", aCone0);
  Bnd_Box aBox0;
  BRepBndLib::Add(aCone0, aBox0);
  printBox(aBox0, "AABB of a simple Cone");
  // MIN: -50.0000001 -100.0000001 -1e-07
  // MAX: 150.0000001 100.0000001 400.0000001
  // DIM: 200.0000002 200.0000002 400.0000002

then define a Cone having the same dimensions, but with a more complex orientation:

  gp_Ax2 aConeAx1(gp_Pnt(100, 200, 300), gp_Dir(1.0, -1.0, 0.0));
  TopoDS_Shape aCone1 = BRepPrim_Cone(aConeAx1, 100, 0, 400).Shell();
  aCone1.Location(aTrans);
  DBRep::Set("c1", aCone1);
  Bnd_Box aBox1;
  BRepBndLib::Add(aCone1, aBox1);
  printBox(aBox1, "AABB of a complex Cone");
  // MIN: 79.2893217813453 -82.842712574619 199.9999999
  // MAX: 432.842712574619 270.710678218655 400.0000001
  // DIM: 353.553390793274 353.553390793274 200.0000002

The bounding boxes do not match, which is expected. Now, lets define transformation gp_Trsf from one coordinate system to another, apply it to our complex-oriented Cone as location TopLoc_Location and compute AABB for it:

  gp_Trsf aTrsf;
  aTrsf.SetTransformation(gp::XOY(), aConeAx1);
  TopoDS_Shape aCone2i = aCone1.Located(aTrsf); // override location
  Bnd_Box aBox2i;
  BRepBndLib::Add(aCone2i, aBox2i);
  printBox(aBox2i, "AABB of a inverse-located Cone (override)");
  // MIN: -100.0000001 -100.0000001 -1e-07
  // MAX: 100.0000001 100.0000001 400.0000001
  // DIM: 200.0000002 200.0000002 400.0000002

We see, that dimensions of this AABB now matches to AABB of trivially-oriented Cone, but min/max is shifted. This is because within TopoDS_Shape::Located() we have overridden original local transformation of the shape. To fix this, we need multiplying transformations:

  TopoDS_Shape aCone2 = aCone1.Located(aCone1.Location() * aTrsf);
  Bnd_Box aBox2;
  BRepBndLib::Add(aCone2, aBox2);
  printBox(aBox2, "AABB of a inverse-located Cone (multiplied)");
  // MIN: -50.0000001 -100.0000001 -1e-07
  // MAX: 150.0000001 100.0000001 400.0000001
  // DIM: 200.0000002 200.0000002 400.0000002

Now we got expected AABB of a Cone within the local coordinate system (in this case, this local coordinate system is where Cone was actually defined - just to be able to compare values).

Instead of computing a new AABB on transformed (located) TopoDS_Shape, we might also apply transformation to AABB Bnd_Box itself:

  Bnd_Box aBox3 = aBox1.Transformed(aTrsf);
  printBox(aBox3, "inverse-located AABB of a Cone");
  // MIN: -285.355339159327 -100.0000001 -64.6446610406726
  // MAX: 214.644661040673 100.0000001 435.355339159327
  // DIM: 500.0000002 200.0000002 500.0000002

Notice that transformed AABB is valid - it contains the entire Cone, - but it is less optimal, as it will grow up while transforming AABB as a simple box. A new AABB computed for transformed TopoDS_Shape will be more tight, as it will consider transformation on-the-fly.

Notice, that in either way BRepBndLib::Add() computes AABB that might be a little bit larger than actual geometry by design, to may calculations reasonably fast. Flag useTriangulation to BRepBndLib::Add() would ask computing AABB on triangulation instead of geometry, when it is available. There are also methods BRepBndLib::AddOptimal() calculating a more precise AABB, and BRepBndLib::AddOBB() calculating an oriented-bounding box instead of axes-aligned one (the latter should more friendly to be transformed).