Strange

Let $a = x^2, b = y^2$ . Sine $a$ and $b$ are both integers, $3a - b$ and $a + b$ must also be integers. So, we find integer solutions to the equation $m^2 + n^2 = 72$ . The only such solutions are $(\pm 6, \pm 6)$ , and we have the system of equations

$\begin{aligned} 3a - b &= \pm 6 \\ a + b &= \pm 6. \end{aligned}$

If we take opposite signs for the 6s in both equations (i.e. -6 and 6), then we find that $a = 0$ , or $x = 0$ , and $x^2 - xy = 0 - 0(y) = 0.$

If we take the negative signs, then we get $a = -3$ , which cannot occur, since $a$ is the square of a real number.

If we take the positive signs, then we get $a = 3$ , or $x = \pm \sqrt{3}$ . We also have $b = 3$ , or $y = \pm \sqrt{3}$ . To maximize the quantity $x^2 - xy$ , we let $x = \sqrt{3}$ and $y = -\sqrt{3}$ , so that

$x^2 - xy = 3 - (\sqrt{3})(-\sqrt{3}) = 3 + 3 = 6.$

Since $6 > 0$ , the maximum value of $x^2 - xy$ is $\boxed{6}$ .

Let positive integers $m$ and $n$ be $m = x^2$ and $n=y^2$ . Then, we have:

$\begin{aligned} (3m-n)^2 + (m+n)^2 & = 72 \\ \Rightarrow 9m^2 - 6mn + n^2 + m^2 + 2mn + n^2 & = 72 \\ 10m^2 - 4mn + 2n^2 & = 72 \\ 5m^2 - 2mn + n^2 & = 36 \\ n^2 - 2mn +5m^2 - 36 & = 0 \\ \Rightarrow n & = \frac{2m \pm \sqrt{4m^2-20m^2+144}}{2} \\ & = m \pm 2\sqrt{9-m^2} \end{aligned}$

For integer $n$ , $\sqrt{9-m^2}$ must be an integer, and there are two possible $m = 0, 3$ and:

$\begin{cases} m = 0 & \Rightarrow n = 6 & \Rightarrow x = 0, y = \pm \sqrt{6} & \Rightarrow x^2 - xy = 0 \\ m = 3 & \Rightarrow n = 3 & \Rightarrow x = \pm \sqrt{3}, y = \pm \sqrt{3} & \Rightarrow x^2 - xy = \begin{cases} (\pm\sqrt{3})^2 - (\pm\sqrt{3})(\pm\sqrt{3}) = 0 \\ (\pm\sqrt{3})^2 - (\color{#D61F06}{\pm}\sqrt{3})(\color{#D61F06}{\mp}\sqrt{3}) = 6 \end{cases} \end{cases}$ .

Therefore, the maximum $x^2 - xy$ is $\boxed{6}$ .