Symmetric Expression with a twist

The expression can be written as

$a^{2}b^{2} -2a^{2}b - 2ab^{2} + 2ab + (a + b -1)^{2}$

Since a + b = 3

$a^{2}b^{2} -2ab( a + b -1) + 4$

$a^{2}b^{2} - 4ab + 4$

Now let $ab = t$

Differentiating the function

$t^{2} - 4t + 4$

we get

2t - 4 = 0

$t = 2$

we have to check if ab = 2 valid ( given a + b = 3)

For (a,b) = {1,2} it satisfies the given condition hence

$ab_{min} = 2$

$a^{2}b^{2} - 4ab + 4 = \boxed{0}$

The expression is equal to $(a-1)^2 ( b-1)^ 2$ .

The minimum is clearly 0, and occurs when $\{ a, b \} = \{ 1, 2 \}$ .

---

Note: Some may think that the minimum occurs when $a = b = \frac{3}{2}$ , which will yield the value 0.0625. Why doesn't this work?

Suppose $n=a+b=3$ and $m=ab$ . Then the given is $m^2-2mn+n^2+2m-2n+1.$ Subbing $n=3$ we end up with $m^2-4m+4=(m-2)^2.$ Since $m\le n^2/4=2.25$ by AM-GM, we can take $m=2$ which yields the minimum, $0$ .

Equality occurs at the roots of $x^2-3x+2$ by Vieta's, which are $1,2$ .

a=1;b=2 then slove

$a^2b^2 -2ab(a+b)+(a+b)^2 -2ab+4ab-2(a+b)+1$ $(ab)^2 -2ab(3)+9+2ab-6+1$ the minimum value of ab >0 is a+b-1=2{note that zero wont work} $2^2 -4\times 3 +9+4+1-6$ $4-12+14-6$ $\boxed{0}$