Croatian Mathematical Olympiad

Let $P(x,y)$ denote plugging $x$ and $y$ into the original equation. We have $$\begin{array}{rl}P(0,1)& :f(0)=0,\\ P(1,1)& :f(f(1))=f(1),\\ P(1,f(1))& :f(f(f(1)))=(1-f(1))f(f(1))+f(1{)}^{2}f(f(1)).\end{array}$$ By combining the second and the third equality we get $$f(1)=(1-f(1))\cdot f(1)+f(1{)}^3,\text{i.e.}f(1{)}^2=f(1{)}^3.$$ From this we have two options: $f(1)=1$ or $f(1)=0$.

If $f(1)=1$, we get $$P(x,1):f(x)=x^2,$$ so $f(x)=x^2$ should hold for all real $x$. By plugging this into the original equation we can conclude that this is not possible. Hence, $f(1)=0$ must hold.

Let $c$ be an arbitrary root of the function $f$. We have $$P(x,c):0=(1-c)f(xc).$$ Therefore, if $c\ne 1$, $f(cx)=0$ must hold for all $x\in \mathbb{R}$. If $c\ne 0$ this implies that $f(x)=0$ for all $x\in \mathbb{R}$. We check directly that $f\equiv 0$ is indeed a possible solution of the original equation.

Now assume that $f$ is not identically equal to 0. Calculations above show in that case 0 and 1 are the only roots of the function $f$. Now we have $$P(1,y):f(f(y))=(1-y)f(y)+y^{2}f(y)=(1-y+y^2)f(y),(1)$$ and for $x\ne 0$ $$P\left(\frac{1}{x},x\right):f\left(\frac{f(x)}{x}\right)=f(x),(2)$$ since $f(1)=0$.

Let $y_1,y_2\in \mathbb{R}\setminus \{0,1\}$ be such that $f(y_1)=f(y_2)\ne 0$. Then from (1) it follows that $$1-y_1+y_1^2=1-y_2+y_2^2,\text{i.e.}(y_1-y_2)(y_1+y_2-1)=0.$$ We conclude that $$f(y_1)=f(y_2)⟹y_1=y_2\text{ or }{y}_1+y_2=1.(3)$$ By combining (2) and (3) we get that for all $x\ne 0,1$ we have $$\frac{f(x)}{x}=x\text{ or }\frac{f(x)}{x}+x=1,$$ i.e. $$f(x)=x^2\text{ or }f(x)=x-x^2.$$ Let $x\in \mathbb{R}\setminus \{0,1\}$ be such that $f(x)=x^2$. By plugging $y=x$ into (1) we get $$f(x^2)=(1-x+x^2)x^2.$$ Now we have two cases: $f(x^2)=x^4$ and $f(x^2)=x^2-x^4$. In the first case, we get $x^4=(1-x+x^2)x^2$, from which we can easily conclude that $x=0$ or $x=1$. Since we assumed $x\notin \{0,1\}$, this is not possible. In the second case, we get $x^2-x^4=(1-x+x^2)x^2$, from which we can conclude $x=0$ or $x=\frac{1}{2}$. Again, we can dismiss $x=0$, and for $x=\frac{1}{2}$ we can note that $f(\frac{1}{2})=(\frac{1}{2}{)}^2=\frac{1}{4}=\frac{1}{2}-(\frac{1}{2}{)}^2$. This shows that $f(x)=x-x^2$ holds for all $x\ne 0,1$. Since this formula holds for $x=0,1$ as well, we can conclude that $$f(x)=x-x^2,\forall x\in \mathbb{R}.$$ By plugging this directly into the original equation, we can confirm that this is indeed a solution. Therefore, the only solutions of the original equation are $f=0,\forall x\in \mathbb{R}$ and $f(x)=x-x^2,\forall x\in \mathbb{R}$.