We factorize it as $xy(x^3+y^3)$ $\Rightarrow xy(x^2+y^2-xy)$ $\Rightarrow xy(1-3xy)$

Now Applying AM-GM inequality.

$\dfrac{3xy+(1-3xy)}{2} \ge \sqrt{3xy(1-3xy)}$

Hence $xy(1-3xy) \le \dfrac{1}{12}$

We set $xy = n$ as $xy$ will be important afterwards. Factorize the original equation gives $xy(x^3+y^3)$ , or, in another form $xy((x+y)^3-3xy(x+y))$ Now replace $xy = n$ and $x+y=1$ , which produce $n(1^3-3n(1))$ . To find the maximum of $x^4y+xy^4$ , you will need to find $n$ which produce the maximum number in $n-3n^2$ . Upon solving the equation, $n$ which produce the maximum value is $n = \frac{1}{6}$ and the equation produced maximum value of $\frac{1}{12}$ , thus, $k=12$

$\begin{aligned} x^4y + xy^4 & = xy(x^3+y^3) \\ & = xy(\color{#3D99F6}{x+y})(\color{#D61F06} {x^2+y^2}-xy) \quad \quad \quad \quad \small \color{#3D99F6}{x+y=1} \\ & = xy(\color{#D61F06}{[x+y]^2 - 2xy} - xy) \quad \quad \quad \quad \small \color{#D61F06}{x^2+y^2=(x+y)^2 -2xy} \\ & = xy(1-3xy) \\ & = - 3(xy)^2 + xy \\ & = - 3 \left((xy)^2 - \frac{1}{3} xy + \frac{1}{36} - \frac{1}{36} \right) \\ & = \frac{1}{12} - \color{#3D99F6}{3 \left(xy - \frac{1}{6}\right)^2} \quad \quad \quad \quad \quad \quad \small \color{#3D99F6}{\text{Since }3 \left(xy - \frac{1}{6}\right)^2 \ge 0} \\ \Rightarrow x^4y + xy^4 & \color{#3D99F6}{\le \boxed{\dfrac{1}{12}}} \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \small \color{#3D99F6}{\text{Note that } xy - \frac{1}{6} = 0, \text{when } x = \frac{1}{2} \pm \frac{\sqrt{3}}{6}, y = \frac{1}{2} \mp \frac{\sqrt{3}}{6}}\end{aligned}$

As both $x$ and $y$ are positive real numbers so

${ sin }^{ 2 }(x)+{ cos }^{ 2 }(x)=1$

Let $x={ sin }^{ 2 }(x)\quad and \quad y={ cos }^{ 2 }(x)$

On putting values we get ${ sin }^{ 8 }(x){ cos }^{ 2 }(x)+{ cos }^{ 8 }(x){ sin }^{ 2 }(x)$

${ sin }^{ 2 }(x){ cos }^{ 2 }(x)\left\{ \left( { cos }^{ 6 }(x)+{ sin }^{ 6 }x \right) \right\}$

Now as ${ ({ sin }^{ 2 }(x)+{ cos }^{ 2 }(x)) }^{ 3 }=1$

${ cos }^{ 6 }(x)+{ sin }^{ 6 }(x)=1-3{ sin }^{ 2 }(x){ cos }^{ 2 }(x)$

Also ${ sin }^{ 2 }(x){ cos }^{ 2 }(x)=\frac { 1 }{ 4 } { sin }^{ 2 }(2x)$

Using these manipulations we get

$\frac { -3 }{ 16 } { sin }^{ 4 }(2x)+\frac { 1 }{ 4 } { sin }^{ 2 }(2x)$

This expression will attain it's maximum value when ${ sin }^{ 2 }(2x)=\frac { 2 }{ 3 }$ and this is in the range of values which $sin^{2}(2x)$ can attain. So on putting values we get it's maximum value as $\frac{1}{12}$ .

Here is the solution :

Let x=cos^2(m) and y=sin^2(m).

required expression =z-3*z^2 . Where z=xy. Hence max value of this parabola is 1/12.

let f= (x^4)y+ x(y^4). we can write f=xy[1 -3xy]. take xy=t. then f= t[1-3t]. for' f ' to be maximum differentiation of 'f' ' w.r.to 't' must be zero. it gives t=1/6. hence xy=1/6. also given x + y=1. solving we get x=1/2 + sqrt(3)/6. and y= 1/2 - sqrt(3)/6. these values of x and Y satisfies given equation. hence f is maximum for xy =1/6. so maximum value of f = 1/6[ 1 - 3*(1/6)]=1/6[ 1 - 1/2]= 1/12. so 1/k = 1/12, hence k= 12.