Simple algebra Problem

Substituting $x={sin^2}u$ and $y={cos}^{2}u$ .WE will get maximum value equal to $6$

From the question, we have $x+y=1\Rightarrow y=1-x$ . Substituting into the second expression, we get: ${ x }^{ 4 }y+x{ y }^{ 4 }=x^{4}(1-x)+x(1-x)^{4}=-3x^{4}+6x^{3}-4x^{2}+x$

Differentiating, we get $\frac{d}{dx}(-3x^{4}+6x^{3}-4x^{2}+x)=-12x^{3}+18x^{2}-8x+1=0$

Solving the cubic equations, we have found 3 solutions for $x$ : $x_{1}=\dfrac{1}{2}, x_{2}=\dfrac{1}{6}(3-\sqrt{3}), x_{3}=\dfrac{1}{6}(3+\sqrt{3})$

Now, we can sub in each of the respective values of $x$ and we get the answer $k=12$ .

$\begin{aligned} x^4y + xy^4 & = xy(x^3+y^3) \\ & = xy{\color{#3D99F6}(x+y)}({\color{#D61F06}x^2+y^2}-xy) & \small \color{#3D99F6} \text{Given that }x+y = 1 \\ & = xy{\color{#3D99F6}(1)}({\color{#D61F06}1-2xy}-xy) & \small \color{#D61F06} \text{Note that }(x+y)^2 = x^2+y^2 + 2xy \\ & = xy - 3(xy)^2 \\ & = - 3\left((xy)^2-\frac 13xy + \frac 1{36}\right) + \frac 1{12} \\ & = \frac 1{12} - \color{#3D99F6}\left(xy - \frac 16\right)^2 & \small \color{#3D99F6} \text{Note that }\left(xy - \frac 16\right)^2 \ge 0 \\ & \le \frac 1{12} \end{aligned}$

$\implies k = \boxed{12}$