An algebra problem by Áron Bán-Szabó

If $a = 1$ or $b = 1$ , then $\frac{1}{a^2} + \frac{1}{ab} + \frac{1}{b^2} > 1.$

Otherwise, both $a \ge 2$ and $b \ge 2$ . In this case, $\frac{1}{a^2} + \frac{1}{ab} + \frac{1}{b^2} \le \frac{1}{4} + \frac{1}{4} + \frac{1}{4} = \frac{3}{4} < 1.$ Therefore, there are no solutions in positive integers.

Let us multiply this equation through by $ab$ to obtain:

$b^2 + ab + a^2 = a^{2}b^{2} \Rightarrow (b^2 - 1)a^2 - ab - b^2 = 0$

which solving for $a$ yields:

$a = \frac{b \pm \sqrt{b^2 - 4(b^2 - 1)(-b^{2})}}{2(b^2 - 1)} = \frac{b \pm \sqrt{4b^4 - 3b^2}}{2(b^2 - 1)} = \frac{b \pm b\sqrt{4b^2 - 3}}{2(b^2 - 1)},$ for $b \ne 1.$

In order for $a$ to be a positive integer, we require the discriminant to be a perfect square as well:

$4b^2 - 3 = k^2$ , for $k \in \mathbb{N}$

or $3 = (2b+k)(2b-k)$ . Since 3 is a prime and is the product of only two positive integers (1 and 3), we require:

$2b + k = 3$ ; $2b - k = 1 \Rightarrow b = k = 1$ . But $b \ne 1$ and it is the only possible solution over the positive integers. Therefore, no positive integer pairs $(a,b)$ exist for this equation.