AM-GM Won't Work Here

For any $n \ge 2$ , let $f_n(x,y,z) = x^ny + y^nz + z^nx$ . We want to maximize $f_n(x,y,z)$ over all nonnegative $x,y,z$ with $x+y+z=1$ . Without loss of generality, we can assume that $x \ge y,z$ . Then $\begin{array}{rcl} \displaystyle f_n(x,y,z) - f_n(x,z,y) & = & x^ny + y^nz + z^nx - x^nz - z^ny - y^nx \\ & = & \displaystyle \left| \begin{array}{ccc} 1 & 1 & 1 \\ x^n & y^n & z^n \\ x & y & z \end{array}\right| \; = \; \left|\begin{array}{ccc} 0 & 0 & 1 \\ x^n - z^n & y^n - z^n & z^n \\ x - z & y - z & z \end{array} \right| \\ & = & \displaystyle (x - z)(y - z)\left[ \sum_{j=0}^{n-1}x^j z^{n-1-j} - \sum_{j=0}^{n-1} y^j z^{n-1-j}\right] \\ & = & \displaystyle (x - z)(y - z) \sum_{j=0}^{n-1} \big(x^j - y^j\big)z^{n-1-j} \\ & = & \displaystyle (x - z)(y - z)(x - y) \sum_{j=1}^{n-1} \sum_{k=0}^{j-1} x^k y^{j-1-k} z^{n-1-j} \end{array}$ and we note that this last double sum is a sum of nonnegative terms (it is the sum of all the monomials in $x,y,z$ of degree $n-2$ ). Thus the largest possible value of $f_n(x,y,z)$ will occur when $x \ge y \ge z$ . But then $\begin{array}{rcl} f_n(x+z,y,0) - f_n(x,y,z) & = & (x+z)^ny - x^ny - y^nz - z^nx \; \ge \; x^ny + nx^{n-1}yz - x^ny - y^nz - z^nx \\ & = & (x^{n-1} - y^{n-1})yz + \big[(n-1)x^{n-2}y - z^{n-1}\big]xz \; \ge \; 0 \end{array}$ so the largest possible value of $f_n(x,y,z)$ occurs when when $x \ge y$ and $z=0$ .

Thus we simply need to maximize $f_n(x,y,0) = x^ny$ subject to the conditions $x,y \ge 0$ and $x+y=1$ . Considering the arithmetic and geometric means of $n$ copies of $\tfrac1nx$ and one copy of $y$ , we deduce that $\left(\frac{x^ny}{n^n}\right)^{\frac1{n+1}} \; \le \; \tfrac1{n+1}\big(n \times \tfrac1nx + y\big) \; = \; \tfrac1{n+1}(x+y) \; = \; \tfrac1{n+1}$ and hence, subject to the condition $x+y = 1$ , $f(x,y,0) \; = \; x^ny \; \le \; \frac{n^n}{(n+1)^{n+1}}$ We note that equality occurs when $x = \tfrac{n}{n+1}$ , $y = \tfrac1{n+1}$ . Thus the maximum value of $f_n(x,y,z)$ over all $x,y,z \ge 0$ with $x+y+z=1$ is $f(\tfrac{n}{n+1},\tfrac1{n+1},0) = \frac{n^n}{(n+1)^{n+1}}$ .

Putting $n=5$ , the maximum value for $f_5(x,y,z)$ , subject to the constraints, is $\frac{5^5}{6^6}$ , which makes the answer to the question $5+5+6+6 = \boxed{22}$ .

Here's the AM - GM solution, first I'll solve the generalized problem. From the condition, we concluded $0\leq x,y,z\leq 1$ . For any real $0\leq a\leq 1$ , we always get $a^n\leq a, n\in\mathbb{Z}$ . If I set $z=min\{x,y,z\}$ , then $x^ny+y^nz+z^nx\leq x^ny+y^nz+zx$ The equality holds when $z=0$ , so now $x^ny$ is what we have left in the $RHS$ , the condition also changed to $x+y=1$ . $1=\frac{x}{n}+\frac{x}{n}+...+\frac{x}{n}+y\stackrel{AM-GM}\geq (n+1)\sqrt[n+1]{\frac{x^ny}{n^n}}$ $\therefore x^ny\leq\frac{n^n}{(n+1)^{n+1}}$ The equality holds when $(x,y,z)=\big(\frac{n}{n+1},\frac{1}{n+1},0\big)$ and its permutations.

Getting back to the main problem, we see $n=5$ so the maximum is $\frac{5^5}{6^6}$ and the answer is $22$

Note : In the application of AM-GM, $\frac{x}{n}$ is repeated $n$ times