Prime Spot

Rewrite as $(x+y)^2-xy = z^2,$ or $(x+y)^2-z^2 = xy.$ This factors: $(x+y-z)(x+y+z) = xy.$ Because $x$ and $y$ are prime, $xy$ has only the following factors: $1,x,y,xy.$

The factor $x+y+z$ is strictly bigger than $x$ and $y,$ so it must be $xy.$ Then $x+y-z$ has to be $1.$ Adding these equations gives $2(x+y)=xy+1,$ which simplifies in the usual way to $(x-2)(y-2)=3.$ The only solutions in prime numbers (indeed, positive integers) $x,y$ are $3,5$ and $5,3,$ which leads to $z=7.$ So the maximum (indeed, the only) product is $xyz=\fbox{105}.$

Relevant wiki: Quadratic Diophantine Equations - Solve by Factoring

Let $f(x,y,z): x^2 + xy + y^2 = z^2$ .

Then $x^2 + xy + (y^2 - z^2) = 0$

The quadratic residue must then be perfect square: $y^2 -4(y^2 - z^2) = n^2$ for some integer $n$ .

Thus, $4z^2 - 3y^2 = n^2$ .

$4z^2 - n^2 = 3y^2$

$(2z-n)(2z+n) = 3y^2$

$$

Since we are looking for prime numbers $x, y, z$ , there are only three possible factorization for $3y^2$ :

1. $3y^2 = 1\times 3y^2$

2. $3y^2 = y\times 3y$

3. $3y^2 = 3\times y^2$

$$

For the first scenario, if $3y^2 = 1\times 3y^2$ , then $2z+n = 3y^2$ and $2z-n = 1$ .

Then $n = 2z - 1$ .

Hence, $4z^2 - n^2 = 4z^2 - (2z-1)^2 = 4z - 1 = 2n + 1 = 3y^2$

$n =\dfrac{3y^2 - 1}{2}$

Then $x = \dfrac{n - y}{2} = \dfrac{3y^2 - 2y -1}{4} = \dfrac{(3y+1)(y-1)}{4}$ .

Since $x$ is prime, there are only three possible factorizations for the right terms:

$$ I. $3y+1 = 4x$ and $y-1 = 1$ . In this case, $y = 2$ , but that will result in $x = \dfrac{7}{4}$ , which is not a prime integer.

II. $3y+1 = x$ and $y-1 = 4$ . In this case, $y = 5$ , but that will result in $x = 16$ , which is not a prime integer.

III. $3y+1 = 2x$ and $y-1 = 2$ . In this case, $y = 3$ , and that leads to $x=5$ and $z^2 = 5^2 + 5\times 3 + 3^2 = 49$ . So $z = 7$ .

Thus, for this scenario, there is only one solution $(5, 3, 7)$ for $f(x,y,z)$ .

$$

Second, for the next scenario, if $3y^2 = y\times 3y$ , then $2z+n = 3y$ and $2z-n = y$ .

Then $4z = 4y$ . $z = y$ , and $n = z = y$ . However, if $n = y$ , $x=0$ , so it is contradicted.

$$

Finally, for the last scenario, if $3y^2 = 3\times y^2$ , then $2z-n = 3$ and $2z+n = y^2$ .

Then $2n = y^2 - 3$ . $n = \dfrac{y^2 -3}{2}$

Thus, $x= \dfrac{n-y}{2} = \dfrac{y^2 - 2y-3}{4} = \dfrac{y^2 - 2y + 1 -4}{4} = (\dfrac{y-1}{2})^2 - 1$

Since $x$ is prime, there is only one possible factorization: $1\times x = (\dfrac{y-1}{2})^2 - 1$

$$

Let $m = \dfrac{y-1}{2}$ for some integer $m$ . Then $m^2 - 1 = (m+1)(m-1) = 1\times x$ .

Thus, $m+1 = x$ , and $m-1 =1$ . Hence, $m = 2$ ; $x=3$ ; and $y = 5$ . And $z=7$ as in the first scenario.

$$ As a result, by checking all possibilities, $(3, 5, 7)$ and $(5, 3, 7)$ the only optimal solutions for $f(x,y,z)$ . Thus, $xyz = \boxed{105}$ .