Your turn again

To determine the 9 elements of the rotation matrix I established an equations system (nonlinear) regarding first the transformation of the given vectors getting 6 equations. Further 6 conditions with respect to properties of the rotation matrix (orthogonality!) completed the set of equations.

A program solving this equations system delivered the matrix elements (ordered by rows):

Lösung im 2. Durchlauf nach 8 Iterationen gefunden: a = -0,259259259259 b = 0,962962962963 c = -0,074074074074 d = -0,518518518519 e = -0,074074074074 f = 0,851851851852 g = 0,814814814815 h = 0,259259259259 i = 0,518518518519

Last but not least the transformation of the interesting vector could be calculated resulting in the solution 26.

Since the rotation $T$ preserves length, angles and orientation, we have the equation $T(\textbf{v}\times\textbf{w})$ $=T(\textbf{v})\times T(\textbf{w})$ for the cross product of any two vectors $\textbf{v}$ and $\textbf{w}$ . For the two given vectors $\textbf{v}=(6,6,3)$ and $\textbf{w}=(-6,3,6)$ we find $T(27,-54,54)=(-63,36,36)$ or, scaled down, $T(3,-6,6)=(-7,4,4)$ . Now $(3,9,12)=\frac{4}{3}(6,6,3)+(-6,3,6)+\frac{1}{3}(3,-6,6)$ , so, by linearity, $T(3,9,12)$ $=\frac{4}{3}(4,-1,8)+(4,8,-1)+\frac{1}{3}(-7,4,4)=(7,8,11)$ . The answer is $\boxed{26}$