China Mathematical Competition

(a) When $a<-2$, then $|f^1(0)|=|a|>2$, therefore $a\notin M$.

(b) When $-2\le a<0$, we have $|f^1(0)|=|a|\le 2$. Assume that $|f^{k-1}(0)|\le |a|\le 2$ for $k\ge 2$. Since $a^2\le -2a$ for $-2\le a<0$, we get that $$-2\le -|a|=a\le f^k(0)=(f^{k-1}(0){)}^2+a\le a^2+a\le |a|\le 2.$$ By the principle of mathematical induction, we conclude that $|f^n(0)|\le a\le 2$ ($\forall n\ge 1$).

(c) When $0\le a\le \frac{1}{4}$, we have $|f^1(0)|=|a|\le \frac{1}{2}$. Assume that $|f^{k-1}(0)|\le \frac{1}{2}$ for $k\ge 2$. We get $$|f^k(0)|\le |f^{k-1}(0){|}^2+a\le {\left(\frac{1}{2}\right)}^2+\frac{1}{4}=\frac{1}{2}.$$ By the principle of mathematical induction, we conclude that $|f^n(0)|\le \frac{1}{2}$ ($\forall n\ge 1$).

From (b) and (c), we obtain $[-2,\frac{1}{4}]\subseteq M$.

(d) When $a>\frac{1}{4}$, define $a_n=f^n(0)$. We have $$a_{n+1}=f^{n+1}(0)=f(f^n(0))=f(a_n)=a_n^2+a,$$ then $a_n>a>\frac{1}{4}$ for any $n\ge 1$. Since $$a_{n+1}-a_n=a_n^2-a_n+a={\left(a_n-\frac{1}{2}\right)}^2+a-\frac{1}{4}\ge a-\frac{1}{4},$$ we get $$\begin{array}{rl}{a}_{n+1}-a& =a_{n+1}-a_1\\ & =(a_{n+1}-a_n)+\cdots +(a_2-a_1)\\ & \ge n\left(a-\frac{1}{4}\right).\end{array}$$ Therefore, when $n>\frac{2-a}{a-\frac{1}{4}}$, we have $$a_{n+1}\ge n\left(a-\frac{1}{4}\right)+a>2-a+a=2.$$ And that means $a\notin M$.

From (a)—(d), we proved that $M=[-2,\frac{1}{4}]$.