For lovers of 3D geometry

First, we'll label the vertices of the solid. The volume of the solid can be split into three parts, one triangular prism in the middle, and two rectangular pyramids on each side (which are mirror images of each other).

All we have to find are the coordinates of point F, and for that, we have to construct the equations of the three planes on which it lies. Assume point $A$ is the origin of the $xy$ plane, then point $B = (12, 0, 0)$ . The angle between the triangular faces of the original pyramid and its base is $\theta_1 = \tan^{-1} \frac{10}{6} = \tan^{-1} \frac{5}{3}$ , while the angle between the cutting plane and the base is $\theta_2 = \frac{1}{2} \theta_1$

Therefore, the equations of the three planes meeting at point $F$ are,

Plane $FAB$ : $(0, - \sin \theta_1, \cos \theta_1) \cdot (x, y, z) = 0$

Plane $FBC$ : $(\sin \theta_1, 0, \cos \theta_1) \cdot (x - 12, y, z ) = 0$

Plane $FAD$ : $(-\sin \theta_2 , 0, \cos \theta_2 ) \cdot (x, y, z) = 0$

Solving the above system of $3$ equations in the $3$ unknowns results in,

$F = (x, y, z) = (8.957, 3.043, 5.071)$

Therfore, the volume is given by

$V = \frac{1}{2} (12) (5.071) (12 - 2(3.043) ) + \frac{2}{3} (12)(3.043)(5.071) = 303.4$

Let $\triangle ABC$ be the cross-section of the pyramid, and label the points and segments as follows:

By the Pythagorean Theorem on $\triangle AMC$ , $l = \sqrt{h^2 + (\frac{s}{2})^2} = \frac{1}{2}\sqrt{4h^2 + s^2}$ .

By the angle bisector theorem, $\frac{AC}{AB} = \frac{CP}{BP}$ , or $\frac{l}{s} = \frac{l - k}{k}$ , which solves to $k = \frac{sl}{s + l}$ .

Since $\triangle BNP \sim \triangle BMC$ by AA similarity, $\frac{NP}{BP} = \frac{CM}{BC}$ , or $\frac{h_2}{k} = \frac{h}{l}$ , which solves to $h_2 = \frac{hk}{l}$ . Also, $\frac{NP}{BN} = \frac{CM}{BM}$ , or $\frac{h_2}{p} = \frac{h}{\frac{s}{2}}$ , which solves to $p = \frac{sh_2}{2h}$ .

Now consider the lower pyramid encased in a right triangular prism:

The volume of the lower pyramid is the volume of the right triangular prism minus twice the volume of the side tetrahedra with perpendicular sides $s$ , $h_2$ , and $p$ , or $V_{\text{pyramid}} = V_{\text{prism}} - 2V_{\text{tetrahedron}} = \frac{1}{2}h_2s^2 - 2 \cdot \frac{1}{6}h_2ps$ , or:

$V_{\text{pyramid}} = \frac{1}{6}h_2s(3s - 2p)$ .

In this question, $s = 12$ and $h = 10$ , so:

$l = \frac{1}{2}\sqrt{4h^2 + s^2} = \frac{1}{2}\sqrt{4 \cdot 10^2 + 12^2} = 2\sqrt{34}$

$k = \frac{ls}{l + s} = \frac{12 \cdot 2\sqrt{34}}{12 + 2\sqrt{34}} = -204 + 36\sqrt{34}$

$h_2 = \frac{hk}{l} = \frac{10(-204 + 36\sqrt{34})}{2\sqrt{34}} = 180 - 30\sqrt{34}$

$p = \frac{12 \cdot (180 - 30\sqrt{34})}{2 \cdot 10} = 108 - 18\sqrt{34}$

$V_{\text{pyramid}} = \frac{1}{6}h_2s(3s - 2p) = \frac{1}{6}(180 - 30\sqrt{34})12(3 \cdot 12 - 2(108 - 18\sqrt{34})) = -138240 + 23760 \sqrt{34} \approx \boxed{303.4}$ .

---

After substituting the above values for $l$ , $k$ , $h_2$ , and $p$ , the general equation for the volume of the pyramid can also be simplified to:

$V_{\text{pyramid}} = \frac{8hs^4(s^2-4h^2)+hs^3(12h^2-s^2)\sqrt{4h^2 + s^2}}{3(4h^2-3s^2)^2}$

in terms of side $s$ and height $h$ .