selecting Face inside another model

Indeed AIS_WireFrame display mode doesn't affect how AIS_Shape is going to be selected - the solid remains selectable as a solid. And this behavior is by design.

I can propose the following approaches:

• Allow user to deactivate selection of particular object. I guess, your application already provides this option.
• Provide user selection filters that would allow to skip outer geometry by some meaningful criteria.
• Allow user to iterate over detected objects under mouse cursor. AIS_InteractiveContext::MoveTo() highlights the topmost object, but it detects all objects below mouse cursor and puts them into sorted list. The highlighting could be switched to the next/previous item via AIS_InteractiveContext::HilightNextDetected() / ::HilightPreviousDetected() methods. In Draw Harness this functionality is mapped to hot-keys '<' and '>', and to commands vselnext/vselprev.
• Override AIS_Shape::ComputeSelection() with desired selection behavior - e.g. by putting only TopoDS_Edge from TopoDS_Face to selection tool StdSelect_BRepSelectionTool, and not TopoDS_Face itself. Or by putting theInteriorFlag=false to StdSelect_BRepSelectionTool::GetSensitiveForFace(). Note that it will be also needed to recompute selection on switching between AIS_Shaded and AIS_WireFrame display modes if you want them behave differently, as selection and display modes live separately within AIS.

  //! Creates the 3D sensitive entities for Face selection.
  //! @param[in]  theFace         face to compute sensitive entities
  //! @param[in]  theOwner        selectable owner object
  //! @param[out] theOutList     output result list to append created entities
  //! @param[in]  theAutoTriang   obsolete flag (has no effect)
  //! @param[in]  theNbPOnEdge    sensitivity parameters
  //! @param[in]  theMaxiParam    sensitivity parameters
  //! @param[in]  theInteriorFlag flag indicating that face interior (TRUE) or face boundary (FALSE) should be selectable
  Standard_EXPORT static Standard_Boolean GetSensitiveForFace (const TopoDS_Face& theFace,
                                                               const Handle(SelectMgr_EntityOwner)& theOwner,
                                                               Select3D_EntitySequence& theOutList,
                                                               const Standard_Boolean theAutoTriang = Standard_True,
                                                               const Standard_Integer theNbPOnEdge = 9,
                                                               const Standard_Real theMaxiParam = 500,
                                                               const Standard_Boolean theInteriorFlag = Standard_True);

pload MODELING VISUALIZATION
box b -100 -100 -100 200 200 200
psphere s 50
vinit View1
vdisplay -dispMode 0 b
vsetcolor b RED
vdisplay -dispMode 1 s
vsetcolor s GREEN
vfit
vmoveto 200 200
vselnext