62nd Ukrainian National Mathematical Olympiad

Answer: $f(x)=x$, $f(x)=0$.

Let us denote the given equation by (1): $$f(x+yf(x+y))=f(y^2)+xf(y)+f(x).(1)$$

Substitute $x=y=0$ into (1): $$f(0)=2f(0)⟹f(0)=0.$$

Substitute $x=0$ into (1): $$f(yf(y))=f(y^2).(2)$$

Substitute $y=-x$ into (1): $$f(x+(-x)f(x-x))=f(x^2)+xf(-x)+f(x)⟹f(x)=f(x^2)+xf(-x)+f(x)$$ So, $$f(x^2)=-xf(-x).(3)$$

Now, substitute $x\to -x$ in (3): $$f(x^2)=-(-x)f(-(-x))=xf(x)$$ So, $$f(x^2)=xf(x)$$ Comparing with (3): $$xf(x)=-xf(-x)⟹f(-x)=-f(x).(4)$$ So $f$ is odd.

Now, substitute $y=-y$ in (1): $$f(x+(-y)f(x-y))=f(y^2)+xf(-y)+f(x)$$ But $f(-y)=-f(y)$, so $$f(x+(-y)f(x-y))=f(y^2)-xf(y)+f(x)$$ Let us write this as: $$f(x+yf(y-x))=f(y^2)-xf(y)+f(x).(5)$$

Now, consider the difference between (1) and (5): $$f(x+yf(x+y))-f(x+yf(y-x))=2xf(y).(6)$$

Suppose that for some $x_0\ne 0$, $f(x_0)=0$. Substitute $y=x_0-x$ into (1): $$\begin{array}{rl}f(x+(x_0-x)f(x+x_0-x))& =f((x_0-x{)}^2)+xf(x_0-x)+f(x)\\ f(x)& =f((x_0-x{)}^2)+xf(x_0-x)+f(x)\\ f((x_0-x{)}^2)& =-xf(x_0-x)\end{array}$$ But from (3): $$f((x_0-x{)}^2)=-(x_0-x)f(x-x_0)=(x_0-x)f(x_0-x)$$ So, $$-xf(x_0-x)=(x_0-x)f(x_0-x)⟹f(x_0-x)=0,\forall x\in \mathbb{R}.$$ Thus, $f$ is identically zero, $f(x)=0$ for all $x$.

If $f$ is not identically zero, then such $x_0\ne 0$ cannot exist. Thus, from the condition $f(x_0)=0$ it follows that $x_0=0$.

Now, let us prove injectivity. Suppose $\exists x_1\ne x_2$ such that $f(x_1)=f(x_2)$. In (6), substitute $x=\frac{x_1-x_2}{2}$, $y=\frac{x_1+x_2}{2}$: $$\begin{array}{rl}f\left(\frac{x_1-x_2}{2}+\frac{x_1+x_2}{2}f(x_1)\right)-f\left(\frac{x_1-x_2}{2}+\frac{x_1+x_2}{2}f(x_2)\right)& =(x_1-x_2)f\left(\frac{x_1+x_2}{2}\right)\\ (x_1-x_2)f\left(\frac{x_1+x_2}{2}\right)& =0\end{array}$$ So, either $x_1=x_2$ (contradicts assumption), or $f\left(\frac{x_1+x_2}{2}\right)=0$. But as above, the only zero is at $0$, so $x_1+x_2=0⟹x_1=-x_2$. But then $$f(x_2)=f(-x_1)=-f(x_1)=-f(x_2)⟹f(x_2)=0⟹x_2=0⟹x_1=0$$ So, $f$ is injective.

Now, from (2): $$f(yf(y))=f(y^2)$$ But $f$ is injective, so $yf(y)=y^2⟹f(y)=y$ for all $y$.

Thus, the only solutions are $f(x)=0$ and $f(x)=x$.

It is easy to check that both satisfy the original equation.