Trigonometry and Algebra Crossover

First, rewrite the system as

$\begin{aligned} x &= xy^2 + 2y \\ y &= yz^2 + 2z \\ z &= zx^2 + 2x. \end{aligned}$

If we solve for $x$ in the first equation, $y$ in the second, and $z$ in the third, we get

$\begin{aligned} x &= \frac{2y}{1 - y^2} \\ y &= \frac{2z}{1 - z^2} \\ z &= \frac{2x}{1 - x^2}. \end{aligned}$

Note that $\tan 2a = \frac{2 \tan a}{1 - \tan ^2 a}$ . Thus, if we let $x = \tan a$ , then $z = \tan 2a$ (third equation), $y = \tan 4a$ (second equation), and $x = \tan 8a$ (first equation).

Since $x = \tan a$ and $x = \tan 8a$ , $\tan a = \tan 8a$ . Since $\tan (b + k\pi) = \tan b$ , for all integers $k$ , we have

$\begin{aligned} \tan (a + k \pi) &= \tan 8a \\ a + k \pi &= 8a \\ 7a &= k \pi \\ a &= \frac{k \pi}{7} . \end{aligned}$

Therefore, $x = \tan \frac{k \pi}{7}$ , for integers $1 \leq k \leq 6$ . (Anything above or below this constraint gives repeated values.)The smallest value of $x$ occurs at $k = 4$ , so our solution is

$(x, y, z) = \left ( \tan \frac{4\pi}{7}, \tan \frac{8\pi}{7}, \tan \frac{16\pi}{7} \right ) = \left ( \tan \frac{4\pi}{7}, \tan \frac{\pi}{7}, \tan \frac{2 \pi}{7} \right ).$

Thus, $x + y + z = \tan \frac{\pi}{7} + \tan \frac{2 \pi}{7} + \tan \frac{4 \pi}{7} = - \sqrt{7}$ , and $a = \boxed{7}$ . (I will post a proof of the final step later; it involves DeMoivre's Theorem.)