Can you solve it geometrically?

First we are going to prove that $a^{2}+b^{2}$ is less than or equal to 40, and second we are going to prove that there is such a polynomial for which the equality can be reached.

Part 1 ( Proof that $a^{2}+b^{2}\le 40$ ): Any 4th degree polynomial satisfying the given condition can be represented in the form of the product of two quadratic polynomials that have discriminants less than or equal to zero, with leading coefficients equal to one, and positive constant terms whose product is one. Then $x^4+ax^3+3x^2+bx+1= (x^2+px+r^2)(x^2+qx+1/r^2)$ where
$p^2\le 4r^2, q^2\le \frac{4}{r^2} \: \:\: \:\:\: (1)$

From the equality by expanding the product and making the coefficients of the same powers of $x$ equal, we also obtain the following equations $p+q=a,\:\;\:\:\: \:\:\:\:\: \:\:\:\:(2)\\ r^{2}+\frac{1}{r^2}+pq=3\:\:\:\:(3)\\ \frac{p}{r^2}+qr^2=b\:\:\:\:\:\:\:\:\:\:(4)$

Multiplying the inequalities in (1) and taking square root of both sides we obtain that $|pq|\le 4$ . Using this inequality and the equality (3) we get $r^{2}+\frac{1}{r^{2}}\le 7$ . Then using the latter inequality and (1), (2), (3) and (4) we obtain the following sequence of inequalities $a^{2}+b^{2}=(p+q)^2+(\frac{p}{r^2}+qr^{2})^2= \\ =p^2+q^2+2pq+\frac{p^2}{r^4}+q^2r^4+2pq\le \\ \le 4r^2+\frac{4}{r^2}+4pq+4r^{2}+\frac{4}{r^2}\le\\ \le 4(r^2+\frac{1}{r^2}+pq+r^{2}+\frac{1}{r^2})\le\\ \le 4(3+7)=40$ Part 2 . (Proof that there is a particular polynomial for which $a^2+b^2=40$ ) Let us consider $r$ as one the solutions of the equation $r^{2}+\frac{1}{r^{2}}=7.$ Then it can be verified by direct calculation that the polynomial $(x^2-2rx+r^2)(x^2+\frac{2}{r}x+\frac{1}{r^2})$ satisfies all the conditions stated. Besides that, by expanding the polynomial and getting the expressions of $a$ and $b$ , we have that $a^{2}+b^{2}= 4(r-1/r)^{2}+4(1/r-r)^{2}=8(r^{2}+\frac{1}{r^2} - 2)= \\=8(7-2)=8*5=40.$

Since $x^4+ax^3+3x^2+bx+1$ is always greater than or equal to 0

$\Rightarrow x^4+ax^3+3x^2+bx+1= (x+r)^4$ or $(x+p)^2(x+q)^2$

Now if desired answer is $(x+r)^4$ then $r=1$ but expanding $(x+1)^4$ we get

$x^4+4x^3+6x^2+4x+1$ which cannot be our solution {as coefficient of $x^2$

are different}

Therefore the only possibility left is $(x+p)^2(x+q)^2$

After expanding and rearranging the coefficients we get

$x^4+2(p+q)x^3+(p^2+q^2+4pq)x^2+2pq(p+q)x+p^2q^2$

Comparing this equation with given equation we can easily see that,

$\boxed{2(p+q)=a}$ ............ $1$

$\boxed{p^2+q^2+4pq=3}$ ............ $2$

$\boxed{2pq(p+q)=b}$ ............ $3$

and $\boxed{p^2q^2=1}$ ............ $4$

Now required value is maximum of all values of $a^2+b^2=4(p+q)^2+4p^2q^2(p+q)^2$

$\Rightarrow$ $ans$ = $max$ { $4(p^2+q^2+2pq)(1+p^2q^2)$ }

Since $p^2q^2=1$ therefore $answer$ is $max$ { $8(p^2+q^2+2pq)$ }

Since $p^2q^2=1$

$\Rightarrow$ $pq=1$ or $-1$ using this in $2^{nd}$ equation we get $p^2+q^2+2pq=1$ or $5$

$\Rightarrow$ $ans$ $=$ $max$ { $8,40$ }

$\Rightarrow$ $\boxed{ans=40}$ .

Sorry but I couldn't find a way to solve it geometrically; I used algebra.

Let $f(x) = x^4 + ax^3 + 3x^2 + bx + 1$ .

Lemma 1: If $a^2+b^2$ is at its maximum value, then $f(x)$ must have a double root.

Proof: Suppose not. Then either it has no real roots or a non-double root. Suppose first that it has no roots. Assume $b \le 0$ (the case for $b > 0$ is similar). Now let $s$ be the minimum value of $\frac{f(x)}{x}$ for $x > 0$ . Then $s > 0$ and $f(x) \ge sx$ for $x > 0$ . Incidentally, for $x \le 0$ we have $f(x) \ge 0 \ge sx$ , so therefore we have $f(x) - sx \ge 0$ for all $x$ . Therefore we have a new function $g(x) = x^4 + ax^3 + 3x^2 + (b-s)x + 1 \ge 0$ , and since $b \le 0$ we have $a^2 + (b-s)^2 > a^2 + b^2$ . This is a contradiction, because $a^2 + b^2$ was supposed to be a maximum.

Therefore $f(x)$ must have a real root. Since $f(x) \ge 0$ , then this root must be a double root $\boxed{}$

Let $m$ be a double root of $f(x)$ . Then, $f(x) = (x-m)^2(x^2+px+q)$ $f(x) = x^4 + (p - 2m)x^3 + (m^2 - 2mp + q)x^2 + (m^2p - 2mq)x + m^2q$ Matching the last coefficient gives $q = \dfrac{1}{m^2}$ . Hence, matching the coefficient of $x^2$ gives $m^2 - 2mp + \dfrac{1}{m^2} = 3$ . Therefore, $p = \dfrac{m^4 - 3m^2 + 1}{2m^3}$ . $(A)$

Now since $f(x) \ge 0$ and $(x-m)^2 \ge 0$ , then $x^2 + px + q$ must also be nonnegative. Hence its discriminant is nonpositive. Thus, $p^2 - 4q \le 0$ $p^2 \le \frac{4}{m^2}$ $|p| \le \frac{2}{|m|}$ $|mp| \le 2 . . . . (B)$ Hence, by equations $(A)$ and $(B)$ , we have $m^4 - 3m^2+1 = 2m^3p$ $|m^4-3m^2+1| = 2m^2|mp| \le 4m^2$ $m^4 - 3m^2+1 \le |m^4 - 3m^2+1| \le 4m^2$ $m^4+1 \le 7m^2 . . . . (C)$ Now we have $a = p - 2m$ and $b = m^2p - \dfrac{2}{m}$ . Therefore, by formulas $(A), (B)$ and $(C)$ we get the following: $a^2+b^2 = (p-2m)^2+\left(m^2p - \frac{2}{m}\right)^2$ $=(1+m^4)p^2 + 4\left(m^2-2mp + \frac{1}{m^2}\right)$ $\le 7m^2p^2 + 4*3 \le 7*2^2 + 12 = 40$ Thus $a^2 + b^2 \le \boxed{40}$ . Note that this value is achieved when $a = 2\sqrt{5}$ and $b = -2\sqrt{5}$ .

This is a result of full numerical analysis simplified with critical facts described as follows:

a and b ought to be of different signs for maximum possible $a^2 + b^2$ . Moreover, their magnitudes ought to be equal. Therefore, we can assign (a = p and b = - p) OR (a = - p and b = p). Found that p = $2 \sqrt5$ . The idea is either a or b helps $x^4 + 3 x^2 + 1$ , while one of the remained to compete against it, so that $a^2 + b^2$ can be maximum.

Maximum $a^2 + b^2$ = 40

$y = x^4 + 3 x^2 + 1$ and $y = a x^3 + b x$ as shown above shall be encountering situations with certain a and b in their geometry. If not because of turning portions to intercept, then the maximum should be infinity. But we want the whole expression to be always NOT smaller than zero. Note that signs between a and b are totally $Interchangable$ !