The answer is 582.

**
This section requires Javascript.
**

You are seeing this because something didn't load right. We suggest you, (a) try
refreshing the page, (b) enabling javascript if it is disabled on your browser and,
finally, (c)
loading the
non-javascript version of this page
. We're sorry about the hassle.

For people who don't know this, it's called Vieta Root Jumping/Root Flipping . I learned it from the second worked example. Which is similar to this problem. (Actually, this problem is very familiar.)

First note that if $a\mid b^{2}-11$ and $b\mid a^{2}-11$ , then $ab\mid a^{2}+b^{2}-11$ . Then let $\dfrac{a^2+b^2-11}{ab}=k$ , for some positive integer $k$ .

$\implies a^2+b^2-11=kab \\ \implies b^2-kab+(a^2-11)=0$

So $b$ is a root of the equation $x^2-kax+(a^2-11)=0.$

By Vieta's Formulas, the other root is $r=(ka-b)=\dfrac{a^2-11}{b}$ . We know that $r$ should be an integer since $b+r=ka$ . In which $b$ and $ka$ are integers.

If $a^2-11>0$ , we have $a^2<ab$ or $a<b$ and $0<r<a$ which follows the conditions. So the solutions $r$ and $a$ here is the same as the solutions $a$ and $b$ . In which they yield the same value of $k$ .

Do what we did earlier again and again until we get a solution $(m,n)$ such that $m^2-11<0$ , we have $m=1,2,3$ .

Case 1: $m=1$

We have $n^2-kn=10$ which yields to the solutions $(n,k)=(2,-3),(5,3),(10,9)$ . In which to the original equation, corresponds to $(a,b)=(1,2),(1,5),(1,10)$ .

The pairs $(1,5)$ and $(1,10)$ have their own recursive sequence of pairs that will satisfy the equation. The pair $(1,5)$ will give rise to the sequence $(a_i,a_{i+1})$ with $a_0=1, a_1=5$ , and the recursion $a_{i+1}=3a_i-a_{i-1}$ for all $i\geq2$ . Similarly, the pair $(1,10)$ will give rise to the sequence $(b_i,b_{i+1})$ with $b_0=1, b_1=10$ , and the recursion $b_{i+1}=9b_i-b_{i-1}$ for all $i\geq2$ .

Case 2: $m=2$

We have $n(n-2k)=7$ which will give rise to the solution $(n,k)=(7,3)$ . In which will correspond to $(a,b)=(2,7)$ . Similar to our first case, this pair will also give rise to a sequence $(c_i,c_{i+1})$ with $c_0=2, c_1=7$ and the recursion $c_{i+1}=3c_i-c_{i-1}$ for all $i\geq2$ .

Case 3: $m=3$

We have $n(n-3k)=2$ . But wait, $n$ should be greater than $m$ , a contradiction. So there are no solutions if $m=3$ .

Solving this took me quite a long time. Plus it would be very long and it would be more understandable if you would refer to this instead.

Summing all solutions, we have $1+238+238+105=\boxed{582}$ .

[Edit] Actually, I first typed in 582. And said I was wrong. So I tried a $+1$ (in case I miscounted something) and got it right. [Edit 2] Oops, so I was right after all