Inspired by Ronak Agarwal

The answer is $\frac{3}{16} \zeta^2 (3) -\frac{\pi^6}{3360}$

$\int_0^{\frac{\pi}{2}}[(\cot x)\log (\sin x) \log^4 (\cos x)]dx=\frac {1}{2^6}\displaystyle\int_0^1 \frac {\log x}{x} \log^4 (1-x)dx$

$\frac {1}{2^6}\displaystyle\int_0^1 \frac {\log x}{x} \log^4 (1-x)dx=\frac {1}{2^6}\displaystyle\int_0^1 \frac {\log (1-x)}{1-x} \log^4 (x)dx$

Let,

$I_s=\displaystyle\int_0^1 \frac {\log (1-x)}{1-x} \log^s (x)dx$

$H_n-H_{n-1}=\frac {1}{n}$

$\sum_{n=1}^{\infty}H_nx^n-x\sum_{n=1}^{\infty}H_{n-1}x^{n-1}=\sum_{n=1}^{\infty}\frac {x^n}{n}$

$\therefore \displaystyle\sum_{n=1}^{\infty}H_nx^n=-\frac{\log (1-x)}{1-x}$

where, $H_0=0$

$I_s=-\displaystyle\sum_{n=1}^{\infty}H_n\int_0^1 x^n \log^s (x)dx$

Now, $\displaystyle\int_0^1 x^n dx=\frac{1}{n+1}$

Differentiating above result with respect to n ,s times,

$\displaystyle\int_0^1 x^n \log^s (x)dx=(-1)^s \frac{s!}{(n+1)^{s+1}}$

$\therefore I_s=(-1)^{s+1}s!\displaystyle\sum_{n=1}^{\infty}\frac{H_n }{(n+1)^{s+1}}$

$I_s=(-1)^{s+1}s!\displaystyle\sum_{n=0}^{\infty}\frac{H_{n+1} }{(n+1)^{s+1}}-\frac{1}{(n+1)^{s+2}}$

$I_s=(-1)^{s+1}s!(\displaystyle\sum_{n=1}^{\infty}\frac{H_{n} }{(n)^{s+1}}-\zeta (s+2))$

Now ,we use ,

$\displaystyle\sum_{n=1}^{\infty}\frac{H_{n} }{(n)^{q}}=(1+\frac{q}{2})\zeta (q+1)-\frac {1}{2}\displaystyle\sum_{k=1}^{q-2}\zeta(k+1)\zeta(q-k)$

Now putting all values together,we get the answer.Proof for above identity can be found here

The solution to this problem is taken from Mr.Tunk-Fey Ariawan's solution to this problem