Prime squares

Let $p$ , $q$ be primes such that $p^2 +7pq +q^2 = m^2$ for some positive integer $m$ .

Then

$5pq = m^2 - (p + q)^2 = (m + p + q)(m- p - q).$

Then $5$ , $p,$ and $q$ divide the product on the right hand side of the equation. We can immediately rule out the possibilities $m+p+q = p, q, 5, 1$ (the case $=5$ is ruled out since $m > p$ , $m > q$ and $p$ , $q$ are at least $2$ ).

Consider the case $m+p+q = 5p$ and $m-p-q = q$ . Eliminating $m$ , we obtain $2(p+q) = 5p - q$ , or $p = q$ .

Similarly, considering the case $m + p + q = 5q$ and $m - p - q = p$ leads to $p = q$ .

Consider the case $m+ p + q = pq$ and $m- p - q = 5$ . Eliminating $m$ , we obtain $2(p + q) = pq - 5$ . This can be reduced to $(p - 2)(q - 2) = 9$ , with solutions $p = q = 5$ or $(p, q) = (3, 11), (11, 3)$ .

Finally consider the case $m + p + q = 5pq$ and $m - p - q = 1$ . Eliminating $m$ , we obtain $2(p+q) = 5pq - 1$ . This can be reduced to $(5p-2)(5q-2) = 9$ , which gives $p = q = 1$ , clearly not prime.

Thus, the set of solutions $(p,q )$ with $p \ne q$ is ${ (3, 11), (11, 3) }.$

Note: This problem is taken from RMO/2001/2 .