Estonian Mathematical Olympiad

Substituting $y=1$ into the given equation, we obtain $$f(0)=f(1)+2(x^2-f(x))(f(1)-1)+1$$ which is equivalent to $$2f(x)(f(1)-1)=f(1)-f(0)+2x^2(f(1)-1).(3)$$

If $f(1)=1$ then (3) implies $0=1-f(0)+1$, or equivalently, $f(0)=2$. Substituting now $y=0$ into the original equation and applying $f(0)=2$ gives us $$f(-2)=2+2(x^2-f(x))+1,$$ or equivalently, $f(x)=x^2+c$ where $c=\frac{3-f(-2)}{2}$. In the other case, $f(x)$ is expressed in the same form with $c=\frac{f(1)-f(0)+1}{2(f(1)-1)}$. We show that the function $f(x)=x^2+c$ satisfies the original equation if and only if $c=1$. Substituting $f(x)=x^2+c$ into the original equation and simplifying leads to $$(c^2-1)y^4+2c(1-c)y^2+(3c^2-2c-1)=0.$$ This equality must hold for every real number $y$. For that, all coefficients in the left hand side must be zeros. From the leading term, we get $c^2-1=0$, implying $c=1$ or $c=-1$. From the quadratic term, we get $2c(1-c)=0$, implying $c=0$ or $c=1$. Altogether, only $c=1$ works. It makes the constant term also zero. Hence $f(x)=x^2+1$ is the only function that satisfies the given equation.

---

Alternative solution.

Substituting $y=0$ into the original equation gives $$f(-f(0))=f(0)+2(x^2-f(x))(f(0)-1)+1,$$ or equivalently, $$2f(x)(f(0)-1)=f(0)-f(-f(0))+1+2x^2(f(0)-1).(4)$$ This implies that either $f(0)=1$ or $f(x)=x^2+\frac{f(0)-f(-f(0))+1}{2(f(0)-1)}$ for any real number $x$.

Consider the case $f(0)=1$. Substituting $x=0$ into the original equation leads to $$f(y^2-1)=f(y^2)-2(f(y)-1)+1,$$ which must hold for any real number $y$. Substituting $-y$ for $y$ gives $$f(y^2-1)=f(y^2)-2(f(-y)-1)+1,$$ which must also hold for any real number $y$. Hence $f(y)=f(-y)$ for any real number $y$, i.e., $f$ is an even function. Eliminating the two terms with $f(0)-1$ in (4) gives $0=1-f(-1)+1$, or equivalently, $f(-1)=2$. Substituting now $y=-1$ into the original equation gives $$f(f(x)-f(-x))=f(1)+2(x^2-f(x))(f(-1)-1)+1.(5)$$ As $f$ is an even function and $f(1)=f(-1)=2$, the equation (5) simplifies to $1=2+2(x^2-f(x))+1$ which is equivalent to $f(x)=x^2+1$.

Hence $f(x)=x^2+c$ for any real number $x$, where $c$ is some constant. We proceed like in Solution 1.