Balkan Mathematical Olympiad Shortlisted Problems

Let $P(x,y)$ denote the given relation. If there is an $a\in \mathbb{R}$ such that $f(a)=0$, then $P(a,y)$ gives that $y=ay+f(a+y)$, and so $f$ must be linear. Then we can easily check and get that the only linear solutions are $f(x)=x$ and $f(x)=2-x$ ($x\in \mathbb{R}$).

Now suppose that $f(x)\ne 0$ for all real numbers $x$. From $P(x-y,y)$ we get that: $$f(x-y+yf(x-y))=-y^2+y(x-1)+f(x).$$ Since $f(t)\ne 0$ for all real numbers $t$, it follows that $-y^2+y(x-1)+f(x)\ne 0$ for all real numbers $x,y$, and so, its discriminant (as a polynomial in $y$) must be negative. That is, $(x-1{)}^2+4f(x)<0$, which gives us $$f(x)<-\frac{(x-1{)}^2}{4}\le 0$$ for all real numbers $x$. Since $(x+1{)}^2\ge 0$ implies that $-\frac{(x-1{)}^2}{4}\le x$, we see that $$f(x)<-\frac{(x-1{)}^2}{4}\le x$$ for all real numbers $x$. Now from $P(x,y)$ for $y>0$ and $x\in \mathbb{R}$, we get that $$xy-y+f(x+y)=f(x+yf(x))<x+yf(x)<x-y\frac{(x-1{)}^2}{4}$$ and so $$f(x+y)<x+y-y(x+\frac{(x-1{)}^2}{4})=x+y-y\frac{(x+1{)}^2}{4}.$$ Setting $x=-y$ above, we get that: $$f(0)<-y\frac{(-y+1{)}^2}{4}.$$ for all positive real numbers $y$. Letting $y\to +\infty$ above, we reach a contradiction. Hence, the only solutions in this functional equation are $f(x)=x$ and $f(x)=2-x$.

---

Alternative solution.

Let $P(x,y)$ denote the given relation. Similarly to the first solution, if a root exists ($f(a)=0$ for any $a$), we get that the function is linear and that the two solutions are $f(x)=x$ and $f(x)=2-x$. Assertion $P(x,c-x)$ gives us the following relation: $$f(x+(c-x)f(x))=(c-x)(x-1)+f(c)=-x^2+(c+1)x+(f(c)-c)$$ The right hand side of the expression is a quadratic equation in $x$ with the discriminant $\Delta =\Delta (c)=(c+1{)}^2+4(f(c)-c)=(c-1{)}^2+4f(c)$. Therefore, if there exists a $c$ such that $(c-1{)}^2+4f(c)\ge 0$, the quadratic equation has a real solution which implies the existence of a root, in which case we are done. If $f(1)=0$, then we found a root and are done. If $f(1)=1$, then by taking $c=1$ we obtain that $\Delta (1)=4$, implying the existence of a root. We now check the case when $f(1)=-1$. From the assertion $P(1-x,x)$, we obtain: $$f(1-x+xf(1-x))=-x^2-1$$ Plugging in $x=1$, in the above assertion, we obtain that $f(f(0))=-2$. Now plugging in $x=1-f(0)$ in the above assertion we get that $f(f(0)+(1-f(0))f(f(0)))=-(1-f(0){)}^2-1$, simplifying and utilizing $f(f(0))=-2$ we obtain $f(3f(0)-2)=-f(0{)}^2+2f(0)-2$. Note that if $f(0)\ge 0$, we have that $\Delta (0)=1+4f(0)>0$, implying the existence of a root, so assume that $f(0)<0$. Now using $c=3f(0)-2$ for our discriminant value, we obtain $\Delta (3f(0)-2)=(3f(0)-3{)}^2+4f(3f(0)-2)=9(f(0)-1{)}^2+4(-f(0{)}^2+2f(0)-2)=5f(0{)}^2-10f(0)+1>0$, implying the existence of a root, and resolving the case when $f(1)=-1$. Now assume that $f(1)\notin \{0,1,-1\}$. From $P(1,y)$, we obtain the relation that $f(1+yf(1))=f(1+y)$. As $f(1)\ne 0$, we can inductively show that $f(1+yf(1{)}^k)=f(1+y)$ for all $k\in \mathbb{Z}$. Since $f(1)\notin \{1,-1\}$, there exists an unbounded sequence $a_n$ such that $f(a_n)$ is constant. Namely, one can take $a_n=1+f(1{)}^{2n}$ if $|f(1)|>1$, and $a_n=1+f(1{)}^{-2n}$ if $|f(1)|<1$, both times it holds that $f(a_n)=f(2)$. The value of the discriminant along this sequence is $\Delta (a_n)=(a_n-1{)}^2+4f(a_n)=(a_n-1{)}^2+4f(2)$, and since $a_n$ is unbounded this there exists $n$ where the value of the discriminant is positive, yielding our root. This finishes the problem.