$All you need is factoring and substitution...$

[(a + b)^2 - 2 ab][(a + b)^3 - 3 a b (a + b)] = 12

=> (1 - 2 a b)(1 - 3 a b) = 12 {since a + b = 1}

=> 6 (a b)^2 - 5 (a b) - 11 = 0

=> (6 a b - 11)(a b + 1) = 0

=> a b = 11/ 6 or -1

With a b = 11/ 6,

a^2 + b^2 = 1 - 2 a b = -8/ 3 {Must not be real a and real b.}

With a b = -1,

a^2 + b^2 = 1 - 2 a b = 3

It is checked that 0.5 [1 + Sqrt (5)] and 0.5 [1 - Sqrt (5)] are the real values for

a^3 + b^3 = 4, a^2 + b^2 = 3 and a + b = 1. Answer is 3.

$Solution:$

We know that $a+b = 1$ so,

$a^2 + b^2 = 1-2ab$

and

$a^3 + b^3 = 1-3ab$

Therefore,

$(1-2ab)(1-3ab) = 12 -> 6(ab)^2 - 5ab - 11 = 0 -> ab = {11/6 , -1}$

Then,

$a^2 + b^2 = 3 or -8/3$

But $a^2 + b^2 = -8/3$ can be obtain if one of a or b is a $imaginary$ $number$ .

So, the acceptable answer is $\boxed{a^2 + b^2 = 3}$ .