SELECTION and TRAINING SESSION

(Solution by A. Asanau, Y. Dubovik, Y. Laurenau, V. Vityaz.) Suppose that such $a$, $b$, $f$ exist. Obviously $b\ne 0$. Further, since $f$ is surjective, there exists a $\lambda$ such that $f(\lambda )=0$. Set $x=\lambda$ in $$f(f(x))=bxf(x)+a,\text{\hspace{1em}(1)}$$ then $f(0)=f(f(\lambda ))=b\lambda f(\lambda )+a=b\lambda \cdot 0+a=a$. Further, $$f(a)=f(f(0))=b\cdot 0\cdot f(0)+a=a.$$ Now $a=f(a)=f(f(a))=baf(a)+a=b{a}^2+a$ whence $a=0$. Thus (1) becomes $$f(f(x))=bxf(x).\text{\hspace{1em}(2)}$$ Further, there exists a $\sigma \ne 0$ such that $f(\sigma )=-1/b$. Set $x=\sigma$ in (2), then $$f(-1/b)=f(f(\sigma ))=b\sigma f(\sigma )=b\sigma (-1/b)=-\sigma .(3)$$ Hence $$f(-\sigma )=f(f(-1/b))=b(-1/b)f(-1/b)=\sigma .$$ Therefore, $$\frac{1}{b}=f(\sigma )=f(f(-\sigma ))=b(-\sigma )f(-\sigma )=-b{\sigma }^2$$ and so $\sigma =\pm 1/b$. If $\sigma =-1/b$, then $\sigma =-1/b=f(\sigma )=f(-1/b)$, contrary to (3). If $\sigma =1/b$, then in view of (3) we have $$\begin{gathered} f(-\sigma )=-\sigma ⟹-\sigma =f(-\sigma )=f(f(-\sigma ))=b(-\sigma )f(-\sigma )=b{\sigma }^2⟹ \\ \sigma =-1/b, \end{gathered}$$ a contradiction.

Therefore, there are no $a$, $b$, $f$ satisfying the problem condition.