Be there or be square

This can be solved through casework. $n$ is a positive integer, so we check the first few possibilities for $n^2$ .

Case 1: $n=1$

Then $a^2b-ab^2=1$ . Then $ab(a-b)=1$ . This implies that both $ab$ and $a-b$ are equal to $1$ (they cannot both be $-1$ as $a$ and $b$ are positive, and so $ab$ is too). In fact, for any $n$ , $ab$ and $a-b$ will have to be factors of $n^2$ . Now, $ab=1$ implies that $a=b=1$ , but then $a-b=0$ , and not $1$ as needed. So the case $n=1$ has no solutions.

Case 2: $n=2$

Then $ab(a-b)=4$ . This implies that pair of numbers $(ab,a-b)$ is equal to either the pair $(1,4)$ or the pair $(2,2)$ . We can analyze these two subcases separately. Again, as before, $ab$ is not equal to $1$ , so say it is equal to $4$ . Then the pair $(a,b)$ is either the pair $(1,4)$ or the pair $(2,2)$ . Neither of these cases satisfy $a-b=1$ . So the pair $(ab,a-b)$ is not the pair $(1,4)$ . If it is the pair $(2,2)$ , then $ab=2$ . Then the pair $(a,b)$ is the pair $(1,2)$ . This doesn't satisfy $a-b=2$ . So the case $n=2$ has no solutions.

Case 3: $n=3$

This case has similar analysis to case 2. Here, $ab(a-b)=9$ . As in case 2, this implies that the pair of numbers $(ab,a-b)$ is equal to either the pair $(1,9)$ or the pair $(3,3)$ . $ab$ is not equal to $1$ , so say it is equal to $9$ . Then the pair $(a,b$ is the pair $(3,3)$ , which does not satisfy $a-b=1$ . So $(ab,a-b)$ is not $(1,9)$ . Then it is the pair $(3,3)$ . Then $ab=3$ . Then the pair $(a,b)$ is the pair $(1,3)$ , which does not satisfy $a-b=3$ . So the case $n=3$ has no solutions.

Case 4: $n=4$

Here, $ab(a-b)=16$ . In this case, there are a few more options for the factors of 16. Doing analysis similar to that in cases 2 and 3, we find that the only solution here is $a=4, b=2$ . So we have a solution of $a=4, b=2, n=4$ . All that is left to do is to prove that this combination minimizes the sum $a+b+n$ .

The solution of $a=4, b=2, n=4$ gives a sum of $a+b+n=10$ . Now, let $n=5+x$ , where $x$ is a non-negative integer. We get to say this, as we have already considered cases where $n<5$ . So now, with $n=5+x$ , we now try to find a solution $a,b,n$ where the sum is less than $10$ . This sum will be $a+b+(5+x)<10$ , or $a+b+x<5$ . Therefore, the sum $a+b$ can be at most $4$ . From this, using the fact that $a>b$ , we get that the only possible remaining option is $a=3,b=1$ . This means that $a^2b-ab^2=6$ . But with $n>4$ , $n^2>16$ , and therefore, $6<n^2$ . Therefore, for $n>4$ , there is no solution $a,b,n$ where $a+b+n$ is less than $10$ . Therefore the solution of $a=4, b=2, n=4$ gives this minimal sum. Then we simply give the answer as $(a+b+n)^2=100$ .

So our final solution is $a=4, b=2, n=4$ with $a+b+n=10$ , and $(a+b+n)^2=100$ .