jmc

First, we claim that any quartic with real coefficients can be written as the product of two quadratic polynomials with real coefficients.

Let $z$ be a complex root of the quartic. If $z$ is not real, then its complex conjugate $\overline{z}$ is also a root. Then the quadratic $(x-z)(x-\overline{z})$ has real coefficients, and when we factor out this quadratic, we are left with a quadratic that also has real coefficients.

If $z$ is real, then we can factor out $x-z,$ leaving us with a cubic with real coefficients. Every cubic with real coefficients has at least one real roots, say $w.$ Then we can factor out $x-w,$ leaving us with a quadratic with real coefficients. The product of this quadratic and $(x-z)(x-w)$ is the original quartic.

So, let $$x^4+a{x}^3+3x^2+bx+1=(x^2+px+r)\left(x^2+qx+\frac{1}{r}\right),(\ast )$$where $p,$ $q,$ and $r$ are real.

Suppose one quadratic factor has distinct real roots, say $z$ and $w.$ Then the only way that the quartic can be nonnegative for all real numbers $x$ is if the roots of the other quadratic are also $z$ and $w.$ Thus, we can write the quadratic as $$(x-z{)}^2(x-w{)}^2.$$Thus, we can assume that for each quadratic factor, the quadratic does not have real, distinct roots. This implies that the discriminant of each quadratic is at most 0. Thus, $$p^2\le 4r\text{and}{q}^2\le \frac{4}{r}.$$It follows that $r>0.$ Multiplying these inequalities, we get $$p^2{q}^2\le 16,$$so $|pq|\le 4.$

Expanding $(\ast )$ and matching coefficients, we get $$\begin{array}{rl}p+q& =a,\\ pq+r+\frac{1}{r}& =3,\\ \frac{p}{r}+qr& =b.\end{array}$$Therefore, $$\begin{array}{rl}{a}^2+b^2& =(p+q{)}^2+{\left(\frac{p}{r}+qr\right)}^2\\ & =p^2+2pq+q^2+\frac{p^2}{r^2}+2pq+q^2{r}^2\\ & =p^2+4pq+q^2+\frac{p^2}{r^2}+q^2{r}^2\\ & \le 4r+4pq+\frac{4}{r}+\frac{4r}{r^2}+\frac{4}{r}\cdot r^2\\ & =4pq+8r+\frac{8}{r}.\end{array}$$From the equation $pq+r+\frac{1}{r}=3,$ $$r+\frac{1}{r}=3-pq,$$so $$a^2+b^2\le 4pq+8(3-pq)=24-4pq\le 40.$$To obtain equality, we must have $pq=-4$ and $r+\frac{1}{r}=7.$ This leads to $r^2-7r+1=0,$ whose roots are real and positive. For either root $r,$ we can set $p=\sqrt{4r}$ and $q=-\sqrt{\frac{4}{r}},$ which shows that equality is possible. For example, we can obtain the quartic $${\left(x-\frac{3+\sqrt{5}}{2}\right)}^2{\left(x+\frac{3-\sqrt{5}}{2}\right)}^2=x^4-2x^3\sqrt{5}+3x^2+2x\sqrt{5}+1.$$Hence, the maximum value of $a^2+b^2$ is $\boxed{40}.$