jmc

Because $g$ has leading coefficient $1$ and roots $r^2,$ $s^2,$ and $t^2,$ we have $$g(x)=(x-r^2)(x-s^2)(x-t^2)$$for all $x.$ In particular, $$\begin{array}{rl}-5=g(-1)& =(-1-r^2)(-1-s^2)(-1-t^2)\\ 5& =(1+r^2)(1+s^2)(1+t^2).\end{array}$$By Vieta's formulas on $f(x),$ we have $r+s+t=-a,$ $rs+st=tr=b,$ and $rst=1.$ Using this, there are two ways to simplify this sum in terms of $a$ and $b$:

First option: Expand and repeatedly apply Vieta. We have $$5=1+(r^2+s^2+t^2)+(r^2{s}^2+s^2{t}^2+t^2{r}^2)+r^2{s}^2{t}^2.$$We immediately have $r^2{s}^2{t}^2=(rst{)}^2=1.$ To get $r^2+s^2+t^2$ in terms of $a$ and $b,$ we write $$r^2+s^2+t^2=(r+s+t{)}^2-2(rs+st+tr)=a^2-2b.$$And to get $r^2{s}^2+s^2{t}^2+t^2{r}^2$ in terms of $a$ and $b,$ we write $$\begin{array}{rl}{r}^2{s}^2+s^2{t}^2+t^2{r}^2& =(rs+st+tr{)}^2-2(r^{2}st+r{s}^{2}t+rs{t}^2)\\ & =(rs+st+tr{)}^2-2rst(r+s+t)=b^2+2a.\end{array}$$Thus, $$5=1+a^2-2b+b^2+2a+1,$$which we can write as $$5=(a+1{)}^2+(b-1{)}^2.$$ Second option: dip into the complex plane. Since $1+z^2=(i-z)(-i-z),$ we can rewrite the equation as $$5=(i-r)(-i-r)(i-s)(-i-s)(i-t)(-i-t).$$Now, for all $x,$ we have $$f(x)=(x-r)(x-s)(x-t),$$so in particular, $f(i)=(i-r)(i-s)(i-t)$ and $f(-i)=(-i-r)(-i-s)(-i-t).$ Thus, $$5=f(i)f(-i).$$We have $f(x)=x^3+a{x}^2+bx-1,$ so $$\begin{array}{rl}5& =(i^3+a{i}^2+bi-1)((-i{)}^3+a(-i{)}^2+b(-i)-1)\\ & =(-(a+1)+(b-1)i)(-(a+1)-(b-1)i),\end{array}$$which simplifies to $$5=(a+1{)}^2+(b-1{)}^2.$$

In either case, the equation we get describes the circle in the $ab-$plane with center $(-1,1)$ and radius $\sqrt{5}.$ It follows that the greatest possible value for $b$ is $\boxed{1+\sqrt{5}}.$