Algebra Craze

Let $m > \sqrt{2}$ .

$\begin{aligned} a+b &=& mx - n \\ ab &=& \dfrac{1}{2}((m^2 - 1)x^2 - mnx - 2n^2) \end{aligned}$

$\implies$

$\begin{aligned} (a+b)^2 &=& m^2x^2 - 2mnx + n^2\\ 2ab &=& (m^2 - 1)x^2 - mnx - 2n^2 \end{aligned}$

$\implies (a + b)^2 - 2ab = a^2 + b^2 = x^2 - mnx + 3n^2$ .

$a = \dfrac{(m^2 - 1)x^2 - mnx - 2n^2}{2b} \implies 2b^2 - 2(mx - n)b + ((m^2 - 1)^2 - mnx - 2n^2) = 0$ which has real value solutions whenever the discriminant $D = 20n^2 - (4m^2 - 8)x^2 \geq 0 \implies |x|^2 \leq \dfrac{5}{m^2 - 2}n^2 \implies |x| \leq \sqrt{\dfrac{5}{m^2 - 2}}n \implies$

$a^2 + b^2 = (\dfrac{5}{m^2 - 2} - \sqrt{\dfrac{5}{m^2 - 2}}m + 3)n^2 = (3 - \sqrt{\dfrac{5}{m^2 - 1}})n^2 \implies \dfrac{5}{m^2 - 2} = 1 \implies m = \sqrt{7}$

$\implies a^2 + b^2 = (4 - \sqrt{7})n^2 \implies$ $\dfrac{a^2 + b^2}{n^2} = 2^2 - \sqrt{7} = \alpha^{\alpha} - \sqrt{\beta} \implies \alpha + \beta = \boxed{9}$ .