The Sum of Square

Let $\sqrt{ab-15}=m$ and $\sqrt{a+b-8} = n$ , where $m$ and $n$ are integers. Then $ab = m^2+15$ and $a+b=n^2+8$ and

$\begin{aligned} a^2b^2 + 16a+16b + 80 & = a^2+32ab+b^2 \\ (ab)^2 + 16(a+b)+80 & = (a+b)^2 +30ab \\ (m^2+15)^2 + 16(n^2+8) + 80 & = (n^2+8)^2 + 30(m^2+15) \\ m^4+30m^2+225 + 16n^2+128 + 80 & = n^4+16n^2+64 + 30m^2+450 \\ m^4 & = n^4 + 3^4 \end{aligned}$

By Fermat last theorem , positive integer solution does not exist for $x^4+y^4=z^4$ . Therefore, $n=0$ and $m=3$ ; and $ab=24$ and $a+b=8$ , $\implies a^2+b^2 = (a+b)^2 - 2ab = 8^2-2(24) = \boxed{16}$ .

$\begin{aligned} a^2b^2+16a+16b+80&=a^2+32ab+b^2\\ a^2b^2-30ab+80&=a^2+2ab+b^2-16(a+b)\\ (ab-15)^2&=(a+b-8)^2+81\\ (ab-15)^2&=(a+b-8)^2+3^4\\ \end{aligned}$ As $\sqrt{ab-15}$ and $\sqrt{a+b-8}$ are integers, ( $ab-15$ ) and ( $a+b-8$ ) are perfect square, which means that $(ab-15)^2$ and $(a+b-8)^2$ are integers power of 4 ( $Exp:0,1,16,81,…$ ). By Fermat Last Theorem, there does not exist postive integer solution for $x^4+y^4=z^4$ . As ( $ab-15$ ) and ( $a+b-8$ ) are nonnegative, we can conclude that $a+b-8=0$ and $(ab-15)^2=3^4$ .

We get ( $a+b=8$ ) and ( $ab=24$ ). $\begin{aligned} a^2+b^2&=(a+b)^2-2ab\\ &=64-2×24\\ &=16 \end{aligned}$ The answer is $\large 16$ .