10^100 is way too high

For people who don't know this, it's called Vieta Root Jumping/Root Flipping . I learned it from the second worked example. Which is similar to this problem. (Actually, this problem is very familiar.)

First note that if $a\mid b^{2}-11$ and $b\mid a^{2}-11$ , then $ab\mid a^{2}+b^{2}-11$ . Then let $\dfrac{a^2+b^2-11}{ab}=k$ , for some positive integer $k$ .

$\implies a^2+b^2-11=kab \\ \implies b^2-kab+(a^2-11)=0$

So $b$ is a root of the equation $x^2-kax+(a^2-11)=0.$

By Vieta's Formulas, the other root is $r=(ka-b)=\dfrac{a^2-11}{b}$ . We know that $r$ should be an integer since $b+r=ka$ . In which $b$ and $ka$ are integers.

If $a^2-11>0$ , we have $a^2<ab$ or $a<b$ and $0<r<a$ which follows the conditions. So the solutions $r$ and $a$ here is the same as the solutions $a$ and $b$ . In which they yield the same value of $k$ .

Do what we did earlier again and again until we get a solution $(m,n)$ such that $m^2-11<0$ , we have $m=1,2,3$ .

Case 1: $m=1$

We have $n^2-kn=10$ which yields to the solutions $(n,k)=(2,-3),(5,3),(10,9)$ . In which to the original equation, corresponds to $(a,b)=(1,2),(1,5),(1,10)$ .

The pairs $(1,5)$ and $(1,10)$ have their own recursive sequence of pairs that will satisfy the equation. The pair $(1,5)$ will give rise to the sequence $(a_i,a_{i+1})$ with $a_0=1, a_1=5$ , and the recursion $a_{i+1}=3a_i-a_{i-1}$ for all $i\geq2$ . Similarly, the pair $(1,10)$ will give rise to the sequence $(b_i,b_{i+1})$ with $b_0=1, b_1=10$ , and the recursion $b_{i+1}=9b_i-b_{i-1}$ for all $i\geq2$ .

Case 2: $m=2$

We have $n(n-2k)=7$ which will give rise to the solution $(n,k)=(7,3)$ . In which will correspond to $(a,b)=(2,7)$ . Similar to our first case, this pair will also give rise to a sequence $(c_i,c_{i+1})$ with $c_0=2, c_1=7$ and the recursion $c_{i+1}=3c_i-c_{i-1}$ for all $i\geq2$ .

Case 3: $m=3$

We have $n(n-3k)=2$ . But wait, $n$ should be greater than $m$ , a contradiction. So there are no solutions if $m=3$ .

Solving this took me quite a long time. Plus it would be very long and it would be more understandable if you would refer to this instead.

Summing all solutions, we have $1+238+238+105=\boxed{582}$ .

[Edit] Actually, I first typed in 582. And said I was wrong. So I tried a $+1$ (in case I miscounted something) and got it right. [Edit 2] Oops, so I was right after all