A calculus problem by Refaat M. Sayed

$\cos (nx) = \begin{cases} \displaystyle \sum_{k=0}^n {n \choose 2k} (-1)^{\left \lfloor \frac n2 \right \rfloor+k}\cos^{2k} x \sin^{n-2k} x & \text{if } n \text{ is even} \\ \displaystyle \sum_{k=0}^n {n \choose 2k+1} (-1)^{\left \lfloor \frac {n+1}2 \right \rfloor+k}\cos^{2k+1} x \sin^{n-2k-1} x & \text{if } n \text{ is odd} \end{cases}$

Considering when $n$ is odd:

$\begin{aligned} I & = \int_0^\frac \pi 2 \tan^m x \cos^{n-2} x \cos nx \ dx \\ & = \int_0^\frac \pi 2 \sin^m x \cos^{n-m-2} x \cos nx \ dx \\ & = \int_0^\frac \pi 2 \sin^m x \cos^{n-m-2} x \sum_{k=0}^n {n \choose 2k+1} (-1)^{\left \lfloor \frac {n+1}2 \right \rfloor+k}\cos^{2k+1} x \sin^{n-2k-1} x \ dx \\ & = \int_0^\frac \pi 2 \sum_{k=0}^n {n \choose 2k+1} (-1)^{\left \lfloor \frac {n+1}2 \right \rfloor + k}\cos^{2k+n-m-1} x \sin^{m+ n-2k-1} x \ dx \\ & = \sum_{k=0}^n {n \choose 2k+1} (-1)^{\left \lfloor \frac {n+1}2 \right \rfloor+k} \int_0^\frac \pi 2 \cos^{2k+n-m-1} x \sin^{m+ n-2k-1} x \ dx \\ & = \frac 12 \sum_{k=0}^n {n \choose 2k+1} (-1)^{\left \lfloor \frac {n+1}2 \right \rfloor+k} B \left( \frac {2k+n-m}2, \frac {m+ n-2k}2 \right) & \small \color{#3D99F6}{\text{There must be a simpler general form.}} \end{aligned}$

Putting $m = 3$ and $n=7$ , we have:

$\begin{aligned} I & = \frac 12 \sum_{k=0}^7 {7 \choose 2k+1} (-1)^{k+1} B \left( k+2, 5-k \right) \\ & = \frac 12 \left(-7B(2,5) + 35B(3.4) - 21B(4,3) + B(5,2) \right) \\ & = 7B(3,4) - 3B(2,5) \\ & = 7 \cdot \frac {\Gamma (3) \Gamma (4)}{\Gamma (7)} - 3 \cdot \frac {\Gamma (2) \Gamma (5)}{\Gamma (7)} \\ & = 7 \cdot \frac {2! 3!}{6!} - 3 \cdot \frac {1! 4!}{6!} \\ & = \frac 7{60} - \frac 1{10} = \frac 1{60} \end{aligned}$

$\implies A+B = 1 + 60 = \boxed{61}$

---

Previous solution

$\begin{aligned} I & = \int_0^\frac \pi 2 \tan^3 x \cos^5 x \color{#3D99F6}{\cos 7x} \ dx \\ & = \int_0^\frac \pi 2 \sin^3 x \cos^2 x \ \color{#3D99F6}{T_7(\cos x)} \ dx & \small \color{#3D99F6}{T_n (\cdot) \text{ is Chebyshev polynomial of the first kind}} \\ & = \int_0^\frac \pi 2 \sin x (1-\cos^2 x) \cos^2 x \ T_7(\cos x) \ dx & \small \color{#3D99F6}{\text{Let }u = \cos x \implies du = - \sin x \ dx} \\ & = \int_0^1 (1-u^2) u^2 \ T_7(u) \ du \\ & = \int_0^1 (1-u^2) u^2 (64u^7-112u^5+56u^3-7u ) \ du \\ & = \int_0^1 (-64u^{11}+176u^9 -168u^7+63u^5-7u^3 ) \ du \\ & = -\frac{64}{12}u^{12}+\frac {176}{10}u^{10} - \frac {168}8u^8 + \frac {63}6u^6 - \frac 74u^4 \ \bigg|_0^1 \\ & = \frac 1{60} \end{aligned}$

$\implies A+B = 1 + 60 = \boxed{61}$

Use property of definite integration f(x)=f(a+b-x)