Non-homogeneous ODE (fixed)

Solving the given differential equation we get $y=93\sin 4t-14\cos 4t+Ae^{-5t}\sin (2t+α)$ , where $A$ and $α$ are constants of integration. Substituting the initial conditions we get $A=2\sqrt 2, α=-\dfrac{π}{4}$ . Hence $y(\dfrac{π}{2})=-14+2e^{-\dfrac{5π}{2}}$ . So $A=2, B=-5$ and $AB=\boxed {-10}$ .

The partial solution to the homogeneous form of the differential equation is given by:

$\begin{aligned} y"+10y' +29y & = 0 \\ (r^2 + 10r+29)e^{rx} & = 0 \\ \implies r & = -5 \pm 2i & \small \blue{\text{where }i=\sqrt{-1} \text{ is the imaginary unit.}} \end{aligned}$

$\begin{aligned} \implies y_1 (x) & = c_1 e^{-5x}\sin 2x + c_2 e^{-5x} \cos 2x & \small \blue{\text{where }c_1 \text{ and } c_2 \text{ are constants.}} \end{aligned}$

Since $y"+10y' +29y = 3538(\sin 2x \cos 2x + \cos^2 2x - \sin^2 2x) = 1769\sin 4x + 3538 \cos 4x$ , assuming the special solution be $y_2(x) = C\sin 4x + D \cos 4x$ , where $C$ and $D$ are constants. Then $y_2' = 4C \cos 4x - 4D \sin 4x$ , $y_2'' = - 16 C\sin 4x - 16 D \cos 4x = -16 y_2$ , and

$\begin{aligned} y_2'' + 10y_2' + 29y_2 & = 1769\sin 4x + 3538 \cos 4x \\ 10(4C\cos 4x - 4D \sin 4x) + 13(C\sin 4x + D\cos 4x) & = 1769\sin 4x + 3538 \cos 4x \end{aligned}$

Equating the coefficients on both sides:

$\begin{cases} 13C - 40D = 1769 \\ 40C + 13D = 3538 \end{cases} \implies C = 93, \ D= -14 \\ \implies y_2(x) = 93 \sin 4x - 14 \cos 4x$

Therefore the general solution of the differential equation is:

$\begin{aligned} y(x) & = c_1 e^{-5x}\sin 2x + c_2 e^{-5x} \cos 2x + 93 \sin 4x - 14 \cos 4x \\ \implies y \left(\frac \pi 2\right) & = -c_2 e^{-\frac {5\pi}2} - 14 \quad \quad \small \blue{\text{As }y(0) = c_2 - 14 = - 16 \implies c_2 = - 2} \\ & = 2e^{-\frac {5\pi}2} - 14 \end{aligned}$

Therefore $AB = (2)(-5) = \boxed{-10}$ .