Problem maximized!

Let us define new variables $X=x+1,Y=y+1,Z=z+1$ to restate the problem as: Find the maximum value of $(XYZ-11)$ when $X+Y+Z=13$ . Using A.M.-G.M. inequality, it works out to $(\frac{13}{3})^3-11=\frac{1900}{27}$ valid for positive values of the variables. Hence answer 1927.

We can solve the problem by breaking the expression into two parts $xyz$ and $xy+yz+zx$ and applying AM-GM inequality . Note that AM-GM inequality only applies if $x,y,z >0$ , which is shown to be the case later.

$\begin{aligned} x+y+z & \ge 3\sqrt[3]{xyz} \\ \implies xyz & \le \left(\frac {x+y+z}3\right)^3 = \frac {1000}{27} \end{aligned}$

Equality occurs when $x=y=z = \frac {10}3 > 0$ .

$\begin{aligned} xy+yz+zx & \le \left(\frac {x+y}2\right)^2 + \left(\frac {y+z}2\right)^2 + \left(\frac {z+x}2\right)^2 \\ & \le \frac 14 \left(x^2+2xy+y^2 + y^2+2yz+z^2 + z^2+2zx+x^2 \right) \\ & \le \frac 12 \left(x^2 + y^2 + z^2 + xy+yz+zx \right) \\ \implies \color{#D61F06}{\frac 32} (xy+yz+zx) & \le \frac 12 \left(x^2 + y^2 + z^2 + \color{#D61F06}{2}(xy+yz+zx) \right) \\ & \le \frac 12 (x+y+z)^2 = 50 \\ \implies xy+yz+zx & \le \frac 23 \cdot 50 = \frac {100}3 \end{aligned}$

Equality again occurs when $x=y=z = \frac {10}3$ . Therefore,

$\implies xyz + xy + yz + zx \le \dfrac {1000}{27} + \dfrac {100}3 = \dfrac {1900}{27}$

$\implies A+Y = 1900 + 27 = \boxed{1927}$

Due to this problem is a calculus problem and the other solutions are algebraic solutions, I'll give a calculus solution

Relevant wiki: Lagrange multipliers

Let $L(x,y,z) = xyz + xy + yz + zx + \lambda(x + y + z - 10) \Rightarrow$ $\dfrac{d}{dx} L(x,y,z) = yz + y + z + \lambda = 0$ $\dfrac{d}{dy} L(x,y,z) = xz + x + z + \lambda = 0$ $\dfrac{d}{dz} L(x,y,z) = yx + y + x + \lambda = 0.$ Substituing $\lambda$ in the previous equations, we get $yz + y + z = xz + x + z = yx + y + x$ $\Rightarrow x = y = z$ and substituing in $x + y + z = 10$ we get $x = y = z = \frac{10}{3}$ .

We can see that the hessian matrix $d^2 L(10/3,10/3,10/3)$ is a defined negative matrix(it's left to the reader as an exercise), so $xyz + xy +yz + zx$ will have a maximum for $x = y = z = \frac{10}{3}$ and this maximum is $(\frac{10}{3})^3 + 3(\frac{10}{3})^2 = \frac{1000}{27} + \frac{100}{3} = \frac{1900}{27}$ ...