Where is the apex?

The algebraic equation of the cone is given by $(\mathbf{r} - \mathbf{r_0} )^T Q (\mathbf{r} - \mathbf{r_0} ) = 0$ , where $\mathbf{r}$ is coordinate vector of any point on the surface of the cone, and $\mathbf{r_0}$ is the coordinate vector of the apex of the cone, and the $3 \times 3$ symmetric matrix matrix $Q$ is given by, $Q = R D R^T$ , with $D$ being a diagonal matrix, given by,

$D = \begin{bmatrix} \cos^2 \theta_c && 0 && 0 \\ 0 && \cos^2 \theta_c && 0 \\ 0 && 0 && -\sin^2 \theta_c \end{bmatrix}$

where $\theta_c$ is the semi-vertical angle. Matrix $R$ is a rotation matrix whose third column is a unit vector that points along the direction of the axis of the cone, while its first two columns are arbitrary unit vectors that form with the third column a valid orthonormal set.

Since the angle $\theta_c$ and the direction of the axis are given, then the matrix $Q$ is fully specified.

Substituting the given points which lie on the surface of the cone into the equation of the cone, we obtain three equations in the unknown vector $\mathbf{r_0}$ ,

$(\mathbf{A} - \mathbf{r_0} )^T Q (\mathbf{A} - \mathbf{r_0} ) = 0 \hspace{12pt} (1)$

$(\mathbf{B} - \mathbf{r_0} )^T Q (\mathbf{B} - \mathbf{r_0} ) = 0\hspace{12pt} (2)$

$(\mathbf{C} - \mathbf{r_0} )^T Q (\mathbf{C} - \mathbf{r_0} ) = 0 \hspace{12pt} (3)$

The second of these equations, can be re-written as follows:

$( (\mathbf{B} - \mathbf{A}) + (\mathbf{A}- \mathbf{r_0}) )^T Q ( (\mathbf{B} - \mathbf{A}) + (\mathbf{A}- \mathbf{r_0}) ) = 0$

And when expanding this, we get,

$(\mathbf{B} - \mathbf{A} )^T Q (\mathbf{B} - \mathbf{A} ) + 2 (\mathbf{B} - \mathbf{A} )^T Q (\mathbf{A} - \mathbf{r_0} )+ (\mathbf{A} - \mathbf{r_0} )^T Q (\mathbf{A} - \mathbf{r_0} ) = 0$

From equation (1), the last term in the above equation is zero, and therefore,

$(\mathbf{B} - \mathbf{A} )^T Q (\mathbf{B} - \mathbf{A} ) + 2 (\mathbf{B} - \mathbf{A} )^T Q (\mathbf{A} - \mathbf{r_0} ) = 0 \hspace{12pt} (4)$

And in a similar fashion, the third equation can be manipulated to yield,

$(\mathbf{C} - \mathbf{A} )^T Q (\mathbf{C} - \mathbf{A} ) + 2 (\mathbf{C} - \mathbf{A} )^T Q (\mathbf{A} - \mathbf{r_0} ) = 0 \hspace{12pt} (5)$

The last two equations are linear equations in the coordinate vector $\mathbf{r_0}$ , and can be readily solved. The solution is of the form,

$\mathbf{r_0} = \mathbf{v_0} + t \mathbf{v_1} \hspace{12pt} (6)$

To determine the value of the parameter $t$ , we plug this expression into any of equations (1) - (3). Plugging it into the first equation, we obtain,

$( (\mathbf{A}-\mathbf{v_0}) - t \mathbf{v_1} )^T Q ( (\mathbf{A}-\mathbf{v_0}) - t \mathbf{v_1} ) = 0 \hspace{12pt} (7)$

This is a quadratic equation in the parameter $t$ , and it has two solutions, one corresponds to the apex vector $\mathbf{r_0}$ that has a positive $z$ -coordinate.

If $D(x, y, z)$ is the the apex of the cone, then the angle between vector $\mathbf{a}$ and $\vec{AD}$ , the angle between vector $\mathbf{a}$ and $\vec{BD}$ , and the angle between vector $\mathbf{a}$ and $\vec{CD}$ must all equal the semi-vertical angle of $\frac{\pi}{12}$ .

Using $\cos \theta = \frac{\mathbf{u} \bullet \mathbf{v}}{|\mathbf{u}||\mathbf{v}|}$ with the above vectors we can obtain the following three equations:

$\cos \frac{\pi}{12} = \frac{\sin \frac{\pi}{10} \cos \frac{2\pi}{3} (x - 5) + \sin \frac{\pi}{10} \sin \frac{2\pi}{3} (y + 2) + \sin \frac{\pi}{10}z}{\sqrt{(x - 5)^2 + (y + 2)^2 + z^2}}$

$\cos \frac{\pi}{12} = \frac{\sin \frac{\pi}{10} \cos \frac{2\pi}{3} (x - 4) + \sin \frac{\pi}{10} \sin \frac{2\pi}{3} (y - 3) + \sin \frac{\pi}{10}z}{\sqrt{(x - 4)^2 + (y - 3)^2 + z^2}}$

$\cos \frac{\pi}{12} = \frac{\sin \frac{\pi}{10} \cos \frac{2\pi}{3} (x + 1) + \sin \frac{\pi}{10} \sin \frac{2\pi}{3} (y - 5) + \sin \frac{\pi}{10}z}{\sqrt{(x + 1)^2 + (y - 5)^2 + z^2}}$

which solves numerically to $(x, y, z) \approx (-3.0924, 5.2345, 17.9836)$ , so $x + y + z \approx \boxed{20.126}$ .

Fundamental unknown quantities:

$\text{Base center coordinates} = \vec{D} = (D_x, D_y, D_z) \\ \text{Base radius} = R \\ \text{Angular parameters associated with intersections} = \alpha, \beta, \gamma$

Processing:
$\text{Axial vector} = \vec{a} \\ \text{Pair of additional unit vectors to make mutually orthogonal set} = \vec{f}, \vec{g} \\ \text{semi-vertical angle} = \text{sva} \\ \text{Cone height} = h = \frac{R}{tan(sva)} \\ \text{Apex location} = \vec{E} = \vec{D} + h \, \vec{a} \\ \text{Base edge point \#1} = \vec{F} = \vec{D} + R \, cos \alpha \, \vec{f} + R \, sin \alpha \, \vec{g} \\ \text{Base edge point \#2} = \vec{G} = \vec{D} + R \, cos \beta \, \vec{f} + R \, sin \beta \, \vec{g} \\ \text{Base edge point \#3} = \vec{H} = \vec{D} + R \, cos \gamma \, \vec{f} + R \, sin \gamma \, \vec{g} \\ \text{Vector \#1 from edge to apex} = \vec{v_F} = \vec{E} - \vec{F} \\ \text{Vector \#2 from edge to apex} = \vec{v_G} = \vec{E} - \vec{G} \\ \text{Vector \#3 from edge to apex} = \vec{v_H} = \vec{E} - \vec{H} \\ \text{Vector weight for xy intersection \#1:} \,\,\,\, F_z + \sigma_F \, v_{Fz} = 0 \implies \sigma_F = -\frac{F_z}{v_{Fz}} \\ \text{Vector weight for xy intersection \#2:} \,\,\,\, G_z + \sigma_G \, v_{Gz} = 0 \implies \sigma_G = -\frac{G_z}{v_{Gz}} \\ \text{Vector weight for xy intersection \#3:} \,\,\,\, H_z + \sigma_H \, v_{Hz} = 0 \implies \sigma_H = -\frac{H_z}{v_{Hz}} \\ \text{xy intersection \#1:} \,\,\,\, \vec{IF} = \vec{F} + \sigma_F \, \vec{v_F} \\ \text{xy intersection \#2:} \,\,\,\, \vec{IG} = \vec{G} + \sigma_G \, \vec{v_G} \\ \text{xy intersection \#3:} \,\,\,\, \vec{IH} = \vec{H} + \sigma_H \, \vec{v_H} \\ \text{Distance to known intersection point \#1} = \Delta A = |\vec{A} - \vec{IF}| \\ \text{Distance to known intersection point \#2} = \Delta B = |\vec{B} - \vec{IG}| \\ \text{Distance to known intersection point \#3} = \Delta C = |\vec{C} - \vec{IH}|$

Performance Criterion: $\Delta A + \Delta B + \Delta C = 0$

Solve for the set of fundamental unknowns so that the performance criterion is satisfied. Doing so results in:

$\text{Apex coordinates} = \vec{E} \approx (-3.093,5.234,17.983)$