The answer is NOT zero.

$\large \displaystyle I = \int_0^{\pi/2} \frac{ \ln(\cos(x)) \ln(\sin(x)) } {\tan(x)} dx$

$\large \displaystyle I = \int_0^{\pi/2} \frac{ \ln(\cos(x)) \ln(\sin(x)) \cos(x)} {\sin(x)} dx$

Make $u = \sin(x)$ . Then, $\cos(x) dx = du$ and $\cos(x) = \sqrt{1 - u^2} = (1-u^2)^{\frac12}$ (the positive square root, since $x \in (0, \pi/2)$ ):

$\large \displaystyle I = \int_0^{1} \frac{ \ln((1 - u^2)^{\frac12}) \ln(u) } {u} dx$

$\large \displaystyle I = \frac12 \int_0^{1} \frac{ \ln(1 - u^2) \ln(u) } {u} dx$

Then, by integration by parts making $f = \ln(1 - u^2)$ and $g' = \frac{\ln(u)}{u}$ :

$\large \displaystyle I = \frac12 \left [ \frac{\ln(1-u^2) \ln^2(u)}{2} \Bigg |_0^1 + \int_0^1 \frac{u \ln^2(u)}{1 - u^2} du \right ]$

The first term goes to $0$ both when $u \rightarrow 0$ and when $u \rightarrow 1$ . This can be shown by the series of $\ln(1-x)$ for $|x| < 1$ (see below). Thus:

$\large \displaystyle I = \frac12 \int_0^1 \frac{u \ln^2(u)}{1 - u^2} du$

We then make $u = e^t$ . Then $du = e^t dt$ and:

$\large \displaystyle I = \frac12 \int_{- \infty}^0 \frac{e^t t^2}{1 - e^{2t}} e^t dt$

Multiplying above and below by $e^{-2t}$ :

$\large \displaystyle I = \frac12 \int_{- \infty}^0 \frac{t^2}{e^{-2t} - 1} dt$ ;

Now we make $t = -\frac{v}{2}$ , $dt = -\frac{dv}{2}$

$\large \displaystyle I = \frac{1}{16} \int_{0}^{\infty} \frac{v^2}{e^v - 1} dv$

By the definition of the Riemann zeta function $\zeta(s)$ :

$\large \displaystyle \zeta(s) = \frac{1}{\Gamma(s)} \int_0^{\infty} \frac{x^{s-1}}{e^x-1} dx$

This integral will be exactly equal to:

$\large \displaystyle I = \frac{1}{16} \Gamma(3) \zeta(3)$

Where $\Gamma(z)$ is the Gamma function . Since $\Gamma(z) = (z-1)!$ :

$\color{#3D99F6} \boxed{\large \displaystyle I = \frac{1}{8}\zeta(3)} \\ \color{#3D99F6} \large \displaystyle a=1, b=8, c = 3 \rightarrow \boxed{\large \displaystyle a+b+c=12}$

---

NOTE: The aforementioned proof. Consider:

$\large \displaystyle C = \ln(1-u^2) \ln^2(u)$

By the Taylor series of $\ln(1-x)$ :

$\large \displaystyle C = - \sum_{k=1}^{\infty} \frac{\ln^2(u) u^{2k}}{k}$

When $u = 1$ , $\ln(u) = 0$ and $C = 0$ .

When $u = 0$ :

$\large \displaystyle C = - \sum_{k=1}^{\infty} \lim_{u \rightarrow 0} \frac{\ln^2(u)}{\frac{k}{u^{2k}}}$

In each term of the sum, we'll have a $\infty / \infty$ case, in which L'Hôpital's rule applies:

$\large \displaystyle C = - \sum_{k=1}^{\infty} \lim_{u \rightarrow 0} \frac{\frac{2\ln(u)}{u}}{\frac{-2k^2}{u^{2k+1}}}$

$\large \displaystyle C = \sum_{k=1}^{\infty} \lim_{u \rightarrow 0} \frac{\ln(u)}{\frac{k^2}{u^{2k}}}$

Again a $\infty / \infty$ case, in which L'Hôpital's rule applies:

$\large \displaystyle C = \sum_{k=1}^{\infty} \lim_{u \rightarrow 0} \frac{\frac{1}{u}}{\frac{-2k^3}{u^{2k+1}}}$ .

$\large \displaystyle C = \sum_{k=1}^{\infty} \lim_{u \rightarrow 0} \frac{-u^{2k}}{2k^3 }$ .

Which will also make $C = 0$ as $u \rightarrow 0$