Inspired by Brian Charlesworth, again

We'll look first at the "bonus" question.

We have $f(x,y,z) = 2axy + 2bxz + 2cyz$ subject to the constraint $g(x,y,z) = x^{2} + y^{2} + z^{2} = 1.$

Setting up (a variation of) the Lagrange equations with $\bigtriangledown f - \lambda \bigtriangledown g = 0,$ we have

• (i) $2ay + 2bz + 2\lambda x = 0 \Longrightarrow -\lambda x + ay + bz = 0,$

• (ii) $2ax + 2cz + 2\lambda y = 0 \Longrightarrow ax - \lambda y + cz = 0,$ and

• (iii) $2bx + 2cy + 2\lambda z = 0 \Longrightarrow bx + cy - \lambda z = 0.$

In matrix form this system is of the form $A \cdot <x,y,z> = <0,0,0>$ where

$A = \begin{bmatrix}-\lambda&a&b\\a&-\lambda&c\\b&c&-\lambda\end{bmatrix}$

Now in order to produce non-zero solutions, $A$ must be singular, i.e., $\det(A) = 0.$ We then have

$\det(A) = -\lambda(\lambda^{2} - c^{2}) + a(bc + a\lambda) + b(ac + b\lambda) = 0 \Longrightarrow \lambda^{3} - (a^{2} + b^{2} + c^{2})\lambda + 2abc = 0,$ (P).

Now as we have learned before from Dr. Bretscher by way of Theorem 3.1 of this link , the maximum $M = \lambda_{1}$ and minimum $m = \lambda_{3}$ are the greatest and least roots of the polynomial $P.$ (Let the "middle" root be $\lambda_{2}$ .)

Now by Vieta's we have that

• $\lambda_{1} + \lambda_{2} + \lambda_{3} = 0,$

• $\lambda_{1} \lambda_{2} + \lambda_{1} \lambda_{3} + \lambda_{2} \lambda_{3} = -(a^{2} + b^{2} + c^{2}),$ and

• $\lambda_{1} \lambda_{2} \lambda_{3} = -2abc.$

From this we see that, since $\lambda_{2} = -\lambda_{1} - \lambda_{2},$ the second of these equations becomes

$-\lambda_{1}(\lambda_{1} + \lambda_{3}) + \lambda_{1} \lambda_{3} - \lambda_{3}(\lambda_{1} + \lambda_{3}) = -(a^{2} + b^{2} + c^{2}) \Longrightarrow \lambda_{1}^{2} + \lambda_{1} \lambda_{3} + \lambda_{3}^{2} = a^{2} + b^{2} + c^{2}.$

So in the way $M$ and $m$ have been designated, we have that $M^{2} + mM + m^{2} = a^{2} + b^{2} + c^{2}.$

In this specific case we have that $a = 5, b = 6$ and $c = 7,$ and so

$M^{2} + mM + m^{2} = 5^{2} + 6^{2}+ 7^{2} = \boxed{110}.$

Brian has written a fine solution. For the sake of variety, I will phrase my solution in the language of quadratic forms, although this is essentially equivalent to Lagrange multipliers.

The symmetric matrix of our quadratic form (from the "bonus question") is $A=\begin{bmatrix}0&a&b\\a&0&c\\b&c&0\end{bmatrix}$

The characteristic polynomial is $p(\lambda)=\lambda^3-(a^2+b^2+c^2)\lambda-2abc$ . Since $M$ and $m$ are roots of $p$ , we have $M^3=(a^2+b^2+c^2)M+2abc$ and $m^3=(a^2+b^2+c^2)m+2abc$ . Subtracting the two equations gives $M^3-m^3=(a^2+b^2+c^2)(M-m)$ so $M^2+Mm+m^2=\frac{M^3-m^3}{M-m}=a^2+b^2+c^2$ .

In the special case $(a,b,c)=(5,6,7)$ the answer is $M^2+Mm+m^2=5^2+6^2+7^2=\boxed{110}$