SAUDI ARABIAN MATHEMATICAL COMPETITIONS

Solution: First, we will prove that $f(x)=0$ has some solutions (maybe not distinct). Indeed, if $f(x)=0$ has no root, then it can be written as $f(x)=(x-c{)}^2+d$ with $d>0$ and $$f(f(x))={\left((x-c{)}^2+d-c\right)}^2+d>0.$$ It means $f(f(x))=0$ has no solution, which is a contradiction.

Now, denote $c_1\ge c_2$ as the solution of $f(x)=0$ and $x_1,x_2$ as the solutions of $f(f(x))=0$ in such a way that $x_1+x_2=-1$. By Vieta's theorem, note that $c_1+c_2=-a$ and $c_1{c}_2=b$.

It is easy to see that $f(f(x))=0$ is equivalent to $f(x)=c_1$, $f(x)=c_2$. We need to consider 2 cases:

1. If $x_1,x_2$ are the solutions of one equation, by Vieta's theorem, then $a=1$. Thus $c_2\le -\frac{1}{2}$.

Consider equation $f(x)-c_2=0$, we have $\Delta =1-4(b-c_2)>0$, which implies that $b<-\frac{1}{4}$.

2. If $x_1,x_2$ are solutions of two equations, then $x_i^2+a{x}_i+b_i=c_i$ with $i=1,2$. Sum these two identities, we get $$x_1^2+x_2^2-a+2b=-a\iff x_1^2+x_2^2+2b=0.$$ Hence, $b=-\frac{x_1^2+x_2^2}{2}\le -\frac{(x_1+x_2{)}^2}{4}=-\frac{1}{4}$.

Therefore, in all case, we always have $b\le -\frac{1}{4}$.