Largest inscribed ellipsoid

As column vectors, we're given that,

$A = \begin{bmatrix} 10 \\ 0 \\ 0 \end{bmatrix}, \hspace{20pt} B = \begin{bmatrix} 5 \\ 12 \\ 0 \end{bmatrix} , \hspace{20pt} C = \begin{bmatrix} -5 \\ 2 \\ 0 \end{bmatrix}, \hspace{20pt} D = \begin{bmatrix} 0 \\ 0 \\ 10 \end{bmatrix}$

These four vertices can be thought of as an affine transformation of the vertices of the a regular tetrahedron. An affine transformation produces an image vector $\mathbf{y}$ of a given vector $\mathbf{x}$ , as follows:

$\mathbf{y = A x + b }$

where $\mathbf{A}$ is $3 \times 3$ and $\mathbf{b}$ is $3 \times 1$ .

An important property of affine transformations is that the volume of an object after the transformation is equal to the absolute value of the determinant $| \mathbf{A} |$ times the its volume before the transformation.

Now, in a regular tetrahedron, the inscribed ellipsoid with maximum volume is the insphere of that tetrahedron. Therefore, for our irregular tetrahedron the inscribed ellipsoid of maximum volume is the image (under the affine transformation which we will determine) of the insphere of the regular tetrahedron. To determine the affine transformation's $\mathbf{A}$ and $\mathbf{b}$ , we can start with any regular tetrahedron. Let's take the tetrahedron whose face radius is $1$ . The edge length is then $\sqrt{3}$ and the height is $\sqrt{2}$ . Therefore, its four vertices can be taken as,

$x_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \hspace{20pt} x_2 = \begin{bmatrix} -\frac{1}{2} \\ \frac{\sqrt{3}}{2} \\ 0 \end{bmatrix} , \hspace{20pt} x_3 = \begin{bmatrix} -\frac{1}{2} \\ -\frac{\sqrt{3}}{2} \\ 0 \end{bmatrix}, \hspace{20pt} x_4 = \begin{bmatrix} 0 \\ 0 \\ \sqrt{2} \end{bmatrix}$

The inradius (the radius of the insphere) of this tetrahedron is one fourth of the height, i.e. $R = \dfrac{\sqrt{2}}{4}$ .

Now, we'll assign the images of $\mathbf{x_1}$ through $\mathbf{x_4}$ to be vertices $A$ through $D$ . Thus,

$\mathbf{y_1} = A , \mathbf{y_2} = B , \mathbf{y_3} = C, \mathbf{y_4} = D$

So that under the affine transformation, $\mathbf{y}_i = \mathbf{A x}_i + \mathbf{b}, \hspace{12pt} i = 1, 2, 3, 4$ .

By augmenting vector $\mathbf{b}$ to matrix $\mathbf{A}$ , we obtain the matrix $\mathbf{M = [A , b ] }$ , and by augmenting $1$ as the fourth entry of vector $\mathbf{x}_i$ , then we can write the following matrix equation:

$\mathbf{Y} = \begin{bmatrix} \mathbf{y}_1 && \mathbf{y}_2 && \mathbf{y}_3 && \mathbf{y}_4 \end{bmatrix} =\mathbf{ M X }$

where $\mathbf{X} = \begin{bmatrix} \mathbf{x}_1 && \mathbf{x}_2 && \mathbf{x}_3 && \mathbf{x}_4 \\ 1 && 1 && 1 && 1 \end{bmatrix}$

Note that $\mathbf{X}$ is a known $4 \times 4$ matrix. Therefore, solving for matrix $\mathbf{M}$ , we obtain,

$\mathbf{M = Y X^{-1} }$

The first three columns of matrix $\mathbf{M}$ is matrix $\mathbf{A}$ , so we can find its determinant, take the absolute value of it and multiply that with the volume of the insphere of the regular tetrahedron, and this gives the desired maximum volume. The answer comes to $\approx \boxed{85.65}$ .