Maximum value

$\begin{aligned} 36(x^4y + xy^4) & = 36xy(x^3+y^3) \\ & = 36xy(\color{#3D99F6}{x+y})(\color{#D61F06}{x^2+y^2}-xy) \quad \quad \small \color{#3D99F6}{\text{Note that: }x+y=1} \\ & = 36xy(\color{#3D99F6}{1})(\color{#D61F06}{(x+y)^2 - 2xy}-xy) \\ & = 36xy(1-3xy) \\ & = -108\left((xy)^2 - \frac13 xy \right) \\ & = -108\left(\left(xy - \frac16\right)^2 - \frac1{36} \right) \\ & = 3 -108 \color{#3D99F6}{\left(xy - \frac16\right)^2} \quad \quad \small \color{#3D99F6}{\text{Since }\left(xy - \frac16\right)^2 \ge 0} \end{aligned}$

$\implies 36(x^4y + xy^4) \le \boxed{3}$

And equality occurs when:

$\begin{aligned} xy & = \frac16 \\ x(1-x) & = \frac16 \\ 6x^2 - 6x + 1 & = 0 \\ \implies x & = \begin{cases} \frac12 + \frac {\sqrt{3}}6 > 0 & \implies y = \frac12 - \frac {\sqrt{3}}6 > 0 \\ \frac12 - \frac {\sqrt{3}}6 > 0 & \implies y = \frac12 + \frac {\sqrt{3}}6 > 0 \end{cases} \end{aligned}$

If we let $x=\frac{1}{2}+h$ and $y=\frac{1}{2}-h$ , then $36xy(x^3+y^3)$ $=\frac{9}{4}(1-4h^2)(12h^2+1)$ $=-108h^4+18h^2+\frac{9}{4}$ $=3-108\left(h^2-\frac{1}{12}\right)^2$ , and the maximum is $\boxed{3}$

The expression's equivalent to $36xy(1-3xy)$ Now we can use AM-GM $3xy(1-3xy)\leq\frac{(1-3xy+3xy)^2}{4}=\frac{1}{4}$ $\Rightarrow 36xy(1-3xy)\leq 3$ The equality happens when $\begin{cases} x+y=1\\xy=\frac{1}{6}\end{cases}$ which implies $x;y$ are roots of the equation $X^2-X+\frac{1}{6}=0$ Therefore $x=\frac{3+\sqrt{3}}{6};y=\frac{3-\sqrt{3}}{6}$ and vice-versa

One can avoid calculations by making substitutions .

Let $x= cos^{2} (\theta )$ , $y=sin^{2}(\theta)$ .

Using trigonometry one gets

$x^{3} + y^{3}= 1-3xy$ .

And also maximum value of a quadratic is $(4ac-b^{2})/4a$ , where $a <0$ .

Now given expression turns out to be

$36 \times (-3(xy)^{2} + xy)$ .

Thus its maximum value is = 36/12=3.

With these types of questions, I always end up relying on calculus:

Let $f(x) = 36(x^4y + xy^4)$

$x+y=1 \implies y = 1-x$

Substitute this into $f(x)$ :

$f(x) = 36\left(x^4(1-x) + x(1-x)^4\right)\\ =36\left(x^4-x^5 + x(1-4x+6x^2-4x^3+x^4)\right)\\ =36\left(x^4-x^5 + x-4x^2+6x^3-4x^4+x^5\right)\\ =36\left(-3x^4+6x^3-4x^2+x\right)$

$f'(x) = 36\left(-12x^3+18x^2-8x+1\right)$

At maximum, minimum points, $f'(x) = 0$

$36\left(-12x^3+18x^2-8x+1\right)=0\\ 12x^3-18x^2+8x-1=0\\ 12x^3-6x^2-12x^2+6x+2x-1=0\\ \left(x-\dfrac{1}{2}\right)\left(12x^2-12x+2\right)=0\\ 2\left(x-\dfrac{1}{2}\right)\left(6x^2-6x+1\right)=0$

$x=\dfrac{1}{2},\;x=\dfrac{6 \pm \sqrt{(6^2) - 4(6)(1)}}{2(6)} = \dfrac{6 \pm \sqrt{12}}{12} = \dfrac{1}{2} \pm \dfrac{1}{\sqrt{12}}$

Next, we find out which points are maximum points with the second derivative test:

$f''(x) = 36\left(-36x^2+36x-8\right) = -144\left(9x^2-9x+2\right)$

$x=\dfrac{1}{2}\implies f''\left(\dfrac{1}{2}\right) = 36 > 0\\ x=\dfrac{1}{2} + \dfrac{1}{\sqrt{12}}\implies f''\left(\dfrac{1}{2}+ \dfrac{1}{\sqrt{12}}\right) = -72 < 0\\ x=\dfrac{1}{2} - \dfrac{1}{\sqrt{12}}\implies f''\left(\dfrac{1}{2}- \dfrac{1}{\sqrt{12}}\right) = -72 < 0$

We have $2$ maximum points, and we calculate the maximum values:

$x=\dfrac{1}{2} + \dfrac{1}{\sqrt{12}}\implies f\left(\dfrac{1}{2}+ \dfrac{1}{\sqrt{12}}\right) = 3\\ x=\dfrac{1}{2} - \dfrac{1}{\sqrt{12}}\implies f\left(\dfrac{1}{2}- \dfrac{1}{\sqrt{12}}\right) =3$

Therefore, the maximum value of $36(x^4y + xy^4)$ is $\boxed{3}$

First,

$x^4y + xy^4 = xy(x+y)((x+y)^2 - 3xy)=xy(1-3xy)$ therefore

$\displaystyle\frac{\mathrm d (x^4y+xy^4)}{\mathrm d (xy)} = 1-6xy$

thus $x^4y + xy^4$ takes maximum when $xy=\frac{1}{6}$ ,

And we get our answer $36 \times \frac{1}{6}\left(1-\frac{1}{2}\right) = \boxed{3}$