Just 3 variables

We can obtain $a^2+b^2$ by writing $a^2+b^2=(a+b)^2-2ab=9x^2-12x+4-8x^2+6x+8=x^2-6x+12=(x-3)^2+3$ , but I tested the answer $3$ , shows me wrong, so I think there must be a restriction about $x$ .

As desired, we can treat $a,b$ like two roots of a quadratic equation $X^2-(a+b)X+ab=0$ , so the discriminant of the equation must be non-negative, so $\begin{aligned}(3x-2)^2-4(4x^2-3x-4)&\geq0\\7x^2&\leq20\\|x|&\leq2\sqrt{\frac57}\end{aligned}$ Substituting $x=2\sqrt{\frac57}$ into the expression gives us the minimum value of $a^2+b^2$ that is $(x-3)^2+3=\bigg(2\sqrt{\frac57}-3\bigg)^2+3=\boxed{4.7152918}$

As others have said, the key insight is the fact that $x$ is bounded in the interval $[\sqrt{-\frac{20}{7}},\sqrt{\frac{20}{7}}]$ . I'll present an alternative way to derive this.

The given equations remind us of the the AM-GM inequality: $\frac{a+b}{2} \geq \sqrt{ab}$

However, as pointed out by Pi Han, we can't technically apply AM-GM here since $a$ and $b$ have to be nonnegative. What we can do, is to manipulate it into the following form which is true for all reals: $\left(\frac{a+b}{2}\right)^2 \geq ab$

Thus, we have $\frac{(3x-2)^2}{4} \geq (4x^2-3x-4)$ This simplifies to: $7x^2-20 \leq 0$ Then we get the desired bounds.

Multiplying the first equation by $a$ ,

$a^2+ab=(3x-2)a$

And substituting the second equation in,

$\begin{aligned}a^2+(4x^2-3x-4)&=(3x-2)a\\ a^2-(3x-2)a+(4x^2-3x-4)&=0\end{aligned}$

We get a quadratic equation. Note that if we were to first multiply by $b$ instead of $a$ we would get the same quadratic, so $a$ and $b$ are the roots of this equation.

Since both $a$ and $b$ have to be real, the discriminant of this equation $\Delta=B^2-4AC$ must be greater than or equal to zero.

$\begin{aligned}\therefore(3x-2)^2-4(4x^2-3x-4)&\geq0\\ 9x^2-12x+4-16x^2+12x+16&\geq0\\ 7x^2&\leq20\\ -\sqrt\frac{20}{7}\leq&x\leq\sqrt\frac{20}{7}\end{aligned}$

We now know that $x$ must lie on the interval $\left[-\sqrt\frac{20}{7},\sqrt\frac{20}{7}\right]$

Now let us get a value for $a^2+b^2$ in terms of x. From the first equation:

$\begin{aligned}(a+b)^2&=(3x-2)^2\\ a^2+b^2+2ab&=9x^2-12x+4\end{aligned}$

Substituting the second equation in,

$\begin{aligned}a^2+b^2+2(4x^2-3x-4)&=9x^2-12x+4\\ a^2+b^2+8x^2-6x-8&=9x^2-12x+4\\ a^2+b^2&=x^2-6x+12\\ a^2+b^2&=(x-3)^2+3\end{aligned}$

The function $y=(x-3)^2+3$ is a concave up parabola with a vertex at $(3,3)$ . Since the vertex is the lowest point and the parabola slopes upwards continuously on either side, the smallest possible value of the function would be at the point closest to the vertex at $x=3$ . This means the minimum value of $a^2+b^2$ occurs when $x=\sqrt\frac{20}{7}$ :

$\begin{aligned}a^2+b^2&=\left(\sqrt\frac{20}{7}-3\right)^2+3\\ &\approx\boxed{4.715}\end{aligned}$

$(a + b)^2 = 9x^2 - 12x + 4$ and $2ab = 8x^2 - 6x - 8$ $\implies (a + b)^2 - 2ab = a^2 + b^2 = x^2 - 6x + 12$ .

$ab = 4x^2 - 3x - 4 \implies a = \dfrac{4x^2 - 3x - 4}{b} \implies b^2 - (3x - 2)b + 4x^2 - 3x - 4 = 0 \implies$

$b = \dfrac{3x - 2 \pm \sqrt{20 - 7x^2}}{2}$ which has real solutions whenever $20 - 7x^2 \geq 0 \implies |x| \leq \sqrt{\dfrac{20}{7}} = 2\sqrt{\dfrac{5}{7}} \implies$

$a^2 + b^2 = \dfrac{20}{7} - 12\sqrt{\dfrac{5}{7}} + 12 \approx \boxed{4.715}$ .

As others have already posted the same solution i was going to post, I'm just here to say I'm glad to be in the 9% who got it right =D