Vietnam Mathematical Olympiad

Suppose that $f:\mathbb{R}\to \mathbb{R}$ satisfies the relation in the problem, i.e. $$f(f(x-y))=f(x)f(y)-f(x)+f(y)-xy,(1)$$ for all $x,y\in \mathbb{R}$. Put $f(0)=a$ By substituting $x=y=0$ into (1) we get $$f(a)=a^2(2)$$ By substituting $x=y=0$ into (1), from (2) we get $$(f(x){)}^2=x^2+a^2,\forall x\in \mathbb{R}(3)$$ Therefore $(f(x){)}^2=(f(-x){)}^2\forall x\in \mathbb{R}$, i.e. $$(f(x)+f(-x))(f(x)-f(-x))=0,\forall x\in \mathbb{R}(4)$$ Suppose that there exist $x_0\ne 0$ such that $f(x_0)=f(-x_0)$. By substituting $y=0$ into (1), we get $$f(f(x))=af(x)-f(x)+a,\forall x\in \mathbb{R}(5)$$ and by substituting $x=0,y=-x$ into (1), we get $$f(f(x))=af(-x)+f(-x)-a,\forall x\in \mathbb{R}.(6)$$ (5) and (6) give $$a(f(x)-f(-x)+f(x)+f(-x)=2a,\forall x\in \mathbb{R}.(7)$$ By substituting $x=x_0$ into (7), we get $$f(x_0)=a(\ast )$$ On the other hand, from (3), we deduce that if $f(x_1)=f(x_2)$ then $x_1^2=x_2^2$. Therefore, (*) implies that $x_0=0$ which contradicts the supposition that $x_0\ne 0$. This contradiction proves that $f(x)\ne f(-x)$, $\forall x\in \mathbb{R}$. Therefore (4) shows that $$f(x)=f(-x)\forall x\ne 0(8)$$ By substituting (8) into (7), we get $$a(f(x)-1)=0\forall x\ne 0,$$ and we deduce from it that $a=0$, since if $a\ne 0$ then $f(x)=1,\forall x\ne 0$ which contradicts (8). So, from (3), we have $$(f(x){)}^2=x^2\forall x\in \mathbb{R}(9)$$ Now suppose that there exists $x_0\ne 0$ such that $f(x_0)=x_0$. Then (5) implies that $$x_0=f(x_0)=-f(f(x_0))=-f(x_0)=-x_0.$$ This contradiction proves that $f(x)\ne x,\forall x\ne 0$. Hence (9) shows that $f(x)=-x,\forall x\in \mathbb{R}$. By direct verification, it is easily seen that this function satisfies the condition of the problem.

So the function $f(x)=-x,\forall x\in \mathbb{R}$, is the unique function satisfying the conditions of the problem.