64 Is Not The Answer!

As this inequality is true for all $(x,y,z)$ , it is possible to divide by $(x+y)(y+z)(z+x)$ . This question becomes equivalent to finding the maxima of $f(x,y,z)=\prod_{cyc} (x^2-xy+y^2)$

$\text{WLOG}$ assume that $z \ge x \ge y$ . Note the following:

$(y+z)^2 \ge y^2-yz+z^2 \text{ } (\because yz \ge 0) \tag{1}$ $x^2 \ge x^2-xy+y^2 \text{ } (\because x \ge y) \tag{2}$

$\text{Also} \text{, } x^2-x(y+z)+(y+z)^2 =x^2-zx+z^2+y(y+2z-x) \ge x^2-zx+z^2 (\because z \ge x) \text{ } \text{(3)}$

If we set $t=6x-x^2$ then $\text{(1)}\times \text{(2)} \times \text{(3)}$ give us

$f(x,y,z) \le f(x,y+z,0)=f(x,6-x,0)=3(6-x)^2x^2(x^2-6x+12)=3t^2(12-t)$ .

And since $t \in [0,9]$ we find that the maximum is $768$ with an equality when $t=8$ .

This is when $x=2, y=0, z=4$ .

Let $x+y+z = S$ and then divide the inequality by $(x+y)(y+z)(z+x)$ to get

$A = (x^2 -xy + y^2)(y^2 -yz + z^2)(z^2 -zx + x^2) \leq k$

So, now we need to maximise A. WLOG , $x \geq y \geq z$

$A = (x^4 - x^3(y+z) + x^2(y^2 + z^2 + zy) -xyz(y+z) + (yz)^2)((z+y)^2 - 3yz)$

Now replace $y+z$ by $S-x$

$A = (x^4 - {x}^{3}(S-x) + {x}^{2}{(S-x)}^{2} + **{yz(yz -Sx)}**)((S-x)^2 - 3yz)$

Now, $yz(yz-Sx) \leq 0$ and the equality holds when $z = 0$ .

Similarly, $(S-x)^2 - 3yz \geq (S - x)^2$ the equality also holding when $z = 0$

So, for a particular $x$ , $A$ is maximum when $z=0$ .

Let $t = x(S-x)$ and $B$ be the maximum value of $A$ for every $x$ .

$B = {t}^{2}[x^2 - x(S-x) + (S-x)^2] = {t}^{2}[S^2 - 3t]$

This is a cubic, and its local maximum occurs at $t = \large\frac{2S^2}{9}$ .

Putting back into the equation, we get $k \geq \large\frac{12S^6}{729} = \boxed{768}$

Also, the equality occurs at $x = \large\frac{2S}{3} = 4 , y = \large\frac{S}{3} = 2 , z = 0$