Less annoying than it looks

$\begin{aligned} I & = \int_0^\frac \pi 2 \sin^{11} x \cos^9 x \ dx \\ & = \int_0^\frac \pi 2 \sin^{11} x \cos^8 x \cdot \cos x\ dx \\ & = \int_0^\frac \pi 2 \sin^{11} x (1-\sin^2 x)^4 \cdot \cos x\ dx & \small \color{#3D99F6} \text{Let }u = \sin x \implies du = \cos x \ dx \\ & = \int_0^1 u^{11}\left(1-u^2\right)^4 \ du \\ & = \int_0^1 \left(u^{19} - 4u^{17} + 6u^{15} - 4u^{13} + u^{11} \right) \ du \\ & = \frac 1{20} - \frac 4{18} + \frac 6{16} - \frac 4{14} + \frac 1{12} \\ & = \frac 1{2520} \end{aligned}$

Therefore, $A+B = 1+2520 = \boxed{2521}$ .

$I=\displaystyle\int_{0}^{\frac{\pi}{2}} \sin^{11}x\cos^{9} dx = \frac{\beta(5,6)}{2} = \frac{4!\cdot 5!}{10!}=\frac{1}{2520}$

$\implies A+B=1+2520=\boxed{2521}$

Try using Walli's formula

$\begin{aligned} \text{Let, I} &= \int_0^{\frac\pi2}\sin^{11}{x}\cos^9{x}\ dx\\ &= \int_0^{\frac\pi2}\cos^{11}{x}\sin^9{x}\ dx \ &\because \int_a^bf(x)\ dx = \int_a^bf(a+b-x)\ dx\\ \text{So, I} &= \frac12 \int_0^{\frac\pi2} \sin^9x\cos^9x\left(\sin^2x+\cos^2x\right)\ dx\\ &= \frac12 \int_0^{\frac\pi2} \sin^9x\cos^9x \ dx & \because \sin^2x+ \cos^2x=1\\ &= \frac12 \int_0^{\frac\pi2} \sin^9{x}\cos{x}{\left(1-\sin^2x\right)}^4\ dx & \text{Let, } u=\sin{x} \implies \ du=\cos{x} \ dx \\ &=\frac12 \int_0^1 {u}^{9}{\left(1-u^2\right)}^4 \ du\\ &= \frac12\int_0^1\left(u^{17} - 4u^{15} + 6u^{13} - 4u^{11} + u^9\right)\ du\\ &= \frac1{2520} \\ &\text{Hence, } A +B = 1 + 2520 = \boxed{2521} \end{aligned}$

$\int \sin ^{11}(x) \cos ^9(x) \, dx \Rightarrow \\ -\frac{63 \cos ^2(x)}{131072}+\frac{21 \cos (4 x)}{1048576}+\frac{7 \cos (6 x)}{131072}+\frac{9 \cos (12 x)}{2097152}+\frac{9 \cos (14 x)}{3670016}+\frac{\cos (20 x)}{10485760}-\frac{3 \cos (8 x)}{262144}-\frac{9 \cos (10 x)}{655360}-\frac{\cos (16 x)}{1048576}-\frac{\cos (18 x)}{4718592}$

How to do the indefinite integral:

Apply this reduction formula five times:

$\int \left(x \cos ^m\right) \left(x \sin ^n\right) \, dx = -\frac{\left(x \cos ^{m+1}\right) \left(x \sin ^{n-1}\right)}{m+n} + \frac{n-1}{m+n} \int \left(x \cos ^m\right) \left(x \sin ^{n-2}\right) \, dx$

giving:

$-\frac{1}{20} \left(x \sin ^{10}\right) \left(x \cos ^{10}\right)-\frac{1}{36} \left(x \sin ^8\right) \left(x \cos ^{10}\right)-\frac{1}{72} \left(x \sin ^6\right) \left(x \cos ^{10}\right)-\frac{1}{168} \left(x \sin ^4\right) \left(x \cos ^{10}\right)-\frac{1}{504} \left(x \sin ^2\right) \left(x \cos ^{10}\right)+\frac{1}{252} \int (x \sin ) \left(x \cos ^9\right) \, dx$

Substitute $u=\cos(x)$ and $du=-sin(x)dx$ : $\frac{1}{252}\,\int u^9\,du \Rightarrow \frac{u^{10}}{10}$ .

All but the last summand of the indefinite integral evaluate to 0 at the integration limits as either the sine or the cosine is 0.

Therefore, evaluating the last term using the change of variable limits $u_{\text{-}}=-\sin(0)=0$ and $u_{\text{+}}=-\sin(\frac{\pi}{2})=-1$ :

$-\frac{1}{252}\times-\frac{1}{10}\Rightarrow \frac{1}{2520}$