Baltic Way shortlist

Answer: The only such a function is $f(x)=0$. At first, assume that $f(x)>0$ for some $x\in \mathbb{R}$. It means that we can choose $y$ such that $$y^2-f(x)=x^{2}y+y$$ (because for $f(x)>0$ this equation has two solutions with respect to $y$), and if we insert it into the given equation we obtain an equality $yf(x{)}^2=0$. As $f(x)>0$ then $y=0$. But $y=0$ is not a solution of $y^2-f(x)=x^{2}y+y$ — contradiction. Thus $f(x)\le 0$ for all $x\in \mathbb{R}$. Note that $f(x)=0$ is a solution. So assume that $f(x_0)<0$ for some $x_0\in \mathbb{R}$. At first we show that $f$ is unbounded. Assume the contrary and put $x_0$ into the equation. We get that $$f(y^2-f(x_0))-f(x_0^{2}y+y)=yf(x_0{)}^2,$$ and see that if $f(x)$ is bounded then the left hand side of this equality also is bounded, but the right hand side is unbounded, that is impossible. If we put $y=0$ in the original equation we get that $f(-f(x))=f(0)$. As $f(x)$ is unbounded and nonpositive we conclude that we can find arbitrarily large $y$ such that $f(y)=f(0)$. Now put $x=x_0$ and choose $y_0$ such that $y_0>\frac{-f(0)}{f(x_0{)}^2}$ and $f(y_0(x_0^2+1))=f(0)$. We get that $$f(y_0^2-f(x_0))=y_{0}f(x_0{)}^2+f(y_0(x_0^2+1))>-f(0)+f(0)=0,$$ what contradicts the fact, that $f(x)\le 0$ for all real $x$.

---

Alternative solution.

For $y=0$ we have $f(-f(x))=f(0)$, in particular $f(-f(0))=f(0)$. Denote $f(0)=c$. For $x=0$ we have $f(y^2-c)=y{c}^2+f(y)$ and substituting $-y$ instead of $y$ gives $$f(y^2-c)=-y{c}^2+f(-y),$$ hence $f(-y)=2y{c}^2+f(y)$ for any $y$. Finally, $$\begin{gathered} yf(x{)}^2+f(x^{2}y+y)=f(y^2-f(x))=f((-y{)}^2-f(x))=-yf(x{)}^2+f(-x^{2}y-y)= \\ =-yf(x{)}^2+2(x^{2}y+y)c^2+f(x^{2}y+y), \end{gathered}$$ hence $2yf(x{)}^2=2(x^{2}y+y)c^2$ and $f(x)=\pm c\sqrt{x^2+1}$ (the choice of $\pm$ may depend on $x$). Then $f(-f(0))=f(0)$ gives $\pm c\sqrt{c^2+1}=c$, $c=0$ and $f(x)=0$ for every $x$.