60th Belarusian Mathematical Olympiad

Suppose that there exists a function $f:\mathbb{R}\to \mathbb{R}$ satisfying $$f(f(x))=1-xf(x)(\ast )$$ for any $x\in \mathbb{R}$.

Set $c=f(0)$. Using $(\ast )$ for $x=0$, we obtain $f(c)=f(f(0))=1-0\cdot f(0)=1-0\cdot c=1$. Since $$1=f(c),(1)$$ we have $f(1)=f(f(c))=1-c\cdot f(c)=1-c\cdot 1=1-c$. Similarly, since $$1-c=f(1),(2)$$ we have $$f(1-c)=f(f(1))=1-f(1)=1-(1-c)=c.(3)$$ It follows that $f(c)=f(f(1-c){)}_{(\ast )}=1-(1-c)\cdot f(1-c{)}_{(3)}=1-(1-c)c=c^2-c+1$, i.e. $$c^2-c+1=f(c).(4)$$ From (1) and (4) it follows that $c^2-c+1=1$, so either $c=0$ or $c=1$. We show that both the equalities $c=0$ and $c=1$ give a contradiction. Indeed, if $c=0$, then from definition of $c$ it follows that $0=f(0)$, but (1) gives $1=f(0)$, a contradiction. If $c=1$, then (1) and (2) give $1=f(1)$ and $0=f(1)$ respectively, a contradiction. Therefore there is no function satisfying $(\ast )$.