Rotation in 3D - Harder Version

First some calculations:
$\$

$\overrightarrow{A{A}'}=\left( \begin{matrix} -1 \\ -1 \\ 5 \\ \end{matrix} \right)$ , $\overrightarrow{B{B}'}=\left( \begin{matrix} 5 \\ 9 \\ 3 \\ \end{matrix} \right)$ . $\$

$\overrightarrow{AC}=\left( \begin{matrix} -3 \\ 1 \\ 5 \\ \end{matrix} \right)\Rightarrow \left| \overrightarrow{AC} \right|=\sqrt{{{\left( -3 \right)}^{2}}+{{1}^{2}}+{{5}^{2}}}=\sqrt{35}$
$\$

$\overrightarrow{BC}=\left( \begin{matrix} -1 \\ 6 \\ 1 \\ \end{matrix} \right)\Rightarrow \left| \overrightarrow{BC} \right|=\sqrt{{{\left( -1 \right)}^{2}}+{{6}^{2}}+{{1}^{2}}}=\sqrt{38}$

$\$

Now, $\overrightarrow {A{A}'}$ and $\overrightarrow {B{B}'}$ are both perpendicular to the axis of rotation, hence, a direction vector of the axis is

$d:= \overrightarrow{A{A}'}\times \overrightarrow{B{B}'}=\left| \begin{matrix} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ -1 & -1 & 5 \\ 5 & 9 & 3 \\ \end{matrix} \right|=\left( \begin{matrix} -48 \\ 28 \\ -4 \\ \end{matrix} \right)$ $\$

Since rotation about an axis is an isometry , we get the following equations:

$\left| \overrightarrow{{A}'{C}'} \right|=\left| \overrightarrow{AC} \right|\Leftrightarrow {{\left| \overrightarrow{{A}'{C}'} \right|}^{2}}={{\left| \overrightarrow{AC} \right|}^{2}}\Leftrightarrow {{\left( x-6 \right)}^{2}}+{{\left( y-3 \right)}^{2}}+\left( z-6 \right)=35 \ \ \ \ \ (1)$ and

$\left| \overrightarrow{{B}'{C}'} \right|=\left| \overrightarrow{BC} \right|\Leftrightarrow {{\left| \overrightarrow{{B}'{C}'} \right|}^{2}}={{\left| \overrightarrow{BC} \right|}^{2}}\Leftrightarrow {{\left( x-10 \right)}^{2}}+{{\left( y-8 \right)}^{2}}+\left( z-8 \right)=38 \ \ \ \ \ (2)$ Furthermore, $\overrightarrow {C{C}'}$ is also perpendicular to $\overrightarrow{d}$ , hence,

$\overrightarrow{C{C}'}\cdot \vec{d}=0\Leftrightarrow -48\left( x-4 \right)+28\left( y-5 \right)-4\left( z-6 \right)=0 \ \ \ \ \ (3)$ Equations $(1)$ , $(2)$ and $(3)$ give two solutions: $\left( x,y,z \right)=\left( \frac{159}{35},\frac{46}{7},\frac{367}{35} \right) \ \ \ \ \ \ \text{and} \ \ \ \ \ \ \left( x,y,z \right)=\left( \frac{55}{9},\frac{74}{9},\frac{29}{9} \right)$ The first solution is rejected, since it leads to a point ${C}'$ for which $\left| \overrightarrow{{A}'{C}'} \right|\ne \left| \overrightarrow{AC} \right|$ , while the second one satisfies the condition $\left| \overrightarrow{{A}'{C}'} \right|= \left| \overrightarrow{AC} \right|$ . Hence, the image of $C$ is ${C}'\left( \frac{55}{9},\frac{74}{9},\frac{29}{9} \right)$ .

For the answer, $x=\dfrac{55}{9}$ , $y=\dfrac{74}{9}$ , $z=\dfrac{29}{9}$ , thus, $9\left( x+y+z \right)=\boxed{158}$ .

The axis has to lie in a plane whose normal is $(A - A')$ and passing through $\frac{1}{2}(A +A')$ , and also in a plane whose normal is $(B - B')$ and passing through $\frac{1}{2} (B + B')$ . Thus the axis line is the intersection of these two planes. The equations are,

$x + y - 5 z = (1, 1, -5) \cdot (6.5, 3.5, 3.5) = -7.5$

$-5 x - 9 y - 3 z = (-5, -9, -3) \cdot (7.5, 3.5, 6.5) = -88.5$

the solution of this system of equations is the line

$(x, y, z) = (-39, 31.5, 0) + t (12, -7, 1)$

so this is our axis of rotation. We still don't know the angle of rotation. So we set up the rotation equation for point A

$A' = p_0 + R (A - p_0)$

here $p_0$ is any point on the axis of rotation, and $R$ is the rotation matrix and is given by

$R = a a^T + (I - a a^T) \cos \theta + S_a \sin \theta \hspace{12pt}(*)$

where $\theta$ is the angle of rotation about the axis whose unit direction vector is $a$ .

$S_a$ is the skew-symmetric matrix which when multiplied by a vector $b$ results in the cross product $a \times b$

Note that all quantities in equation $(*)$ are known except $\theta$ .

Plugging in $a = \dfrac{(12, -7, 1)}{ \sqrt{ 12^2 + (-7)^2 + 1^2 } }$ and $p_0 = (-39, 31.5, 0)$ , and the coordinates of point $A$ , we obtain

$(6, 3, 6) = v_0 + v_1 \cos \theta + v_2 \sin \theta$

since $v_1$ and $v_2$ are perpendicular to each other, then

$\cos \theta =\dfrac{ ( (6, 3, 6) - v_0 ) \cdot v_1 }{ v_1 \cdot v_1 }$

$\sin \theta = \dfrac{ ( (6, 3, 6) - v_0 ) \cdot v_2 }{ v_2 \cdot v_2 }$

and from this we can calculate $\theta \approx -2.210188$

Now the rotation is fully specified.

Now the image of point $C$ is given by,

$C' = p_0 + R ( C - p_0)$

performing the calculations, we get $C' = ( 6 \frac{1}{9} , 8 \frac{2}{9} , 3 \frac{2}{9} )$

so that $9(x + y + z) = \boxed{158}$ .