Think Positive

In order to find the desired integral solution, we may very well put this system into x-y coordinates:

$x + 2y -12 > 0 \Rightarrow y > \dfrac{-x}{2} + 6$ $3x - y -17 > 0 \Rightarrow y < 3x - 17$ $80 - 9x - 4y > 0 \Rightarrow y < \dfrac{-9x}{4} + 20$

Now we can plot the linear graphs and label the inequality zone as shown below:

For $y > \dfrac{-x}{2} + 6$ (red), the solution lies above the red line as the higher y-value suffices the inequality.

Then for $y < 3x - 17$ (blue), the solution lies rightward to the line as the higher x-value suffices the inequality.

Finally, for $y < \dfrac{-9x}{4} + 20$ (green), the solution lies leftward to the green line as the lower x-value suffices the inequality.

To summarize, the solution point lies within the yellow triangular zone.

Since the red line intersects the blue and the green at points $(\dfrac{46}{7} , \dfrac{19}{7})$ and $(8 , 2 )$ respectively, then $\dfrac{46}{7} < x < 8$ . Thus, the desired $x = 7$ . $\boxed{a = 7}$ .

And when $x = 7$ , the point at the blue line is $(7 , 4)$ . With the previous intersected point $(8 , 2)$ , we can conclude that $2 < y < 4$ . Thus, the desired $y = 3$ . $\boxed{b = 3}$ .

As a result, $a^2 + b^2 = 7^2 + 3^2 = \boxed{58}$