Inspired By Christian Lim

Given that

$\begin{aligned} (1+2x)(1-2x) & = y(5y+8x) \\ 1-4x^2 & = 5y^2 + 8xy \\ 4x^2 +8xy + 5y^2 & = 1 \\ 4x^2 +8xy + 4y^2 + y^2 & = 1 \\ (2(x+y))^2 + y^2 & = 1 \end{aligned}$

The given equation is an ellipse and we can substitute

$\begin{cases} 2(x+y) = \cos \theta \\ \sin \theta = \sin \theta \end{cases} \implies x = \frac 12 \cos \theta - \sin \theta$

Now we have:

$\begin{aligned} 2x^2 + 3xy + 2y^2 & = 2(x+y)^2 - xy \\ & = \frac 12 \cos^2 \theta - \left(\frac 12 \cos \theta - \sin \theta \right) \sin \theta \\ & = \frac 12 \cos^2 \theta + \sin^2 \theta - \frac 12 \sin \theta \cos \theta \\ & = \frac 12 + \frac 12 \sin^2 \theta - \frac 12 \sin \theta \cos \theta \\ & = \frac 12 + \frac 14 \left(1- \cos 2 \theta\right) - \frac 14 \sin 2 \theta \\ & = \frac 34 - \frac {\sqrt 2}4 \sin \left(2\theta + \frac \pi 4\right) \end{aligned}$

Therefore,

$\implies \begin{cases} M = \max \left(\frac 34 - \frac {\sqrt 2}4 \sin \left(2\theta + \frac \pi 4\right) \right) = \frac {3+\sqrt 2}4 & \text{when } \sin \left(2\theta + \frac \pi 4\right) = - 1 \\ N = \min \left(\frac 34 - \frac {\sqrt 2}4 \sin \left(2\theta + \frac \pi 4\right) \right) = \frac {3-\sqrt 2}4 & \text{when } \sin \left(2\theta + \frac \pi 4\right) = 1 \end{cases}$

$\implies M+N+MN = \frac {3+\sqrt 2}4 + \frac {3-\sqrt 2}4 + \frac {3+\sqrt 2}4 \times \frac {3-\sqrt 2}4 = \frac 32 + \frac 7{16} = \frac {31}{16}$ $\implies a+b = 31+16 = \boxed{47}$ .