Estonian Mathematical Olympiad

Note that if $x\ne 0$ and $x\ne 1$ then $f(x)\ne 0$ and $f(x)\ne 1$. Indeed, difference from 0 is given in the problem and, if we supposed $f(x)=1$, taking $y=1$ would yield $f(f(xy))=f(f(x))=f(1)$, meaning that $f(f(xy))$ were undefined and could not satisfy the equation given in the problem. Hence one can apply $f$ infinitely to any real number $x\notin \{0,1\}$.

Plugging $x=2,y=\frac{z}{2}$, where $z\notin \{0,1\}$, into the original equation yields $$f(f(z))=1-\frac{2}{zf(f(f(2)))}.$$ We see that $f(f(z))$ obtains all real values except 1 and $1-\frac{2}{f(f(f(2)))}$, when $z$ obtains all real values except 0 and 1. As $f(f(z))$ cannot obtain the value 0, the only option is $1-\frac{2}{f(f(f(2)))}=0$, i.e., $f(f(z))$ obtains all real values except 0 and 1.

Now substituting $x=z$ and $y=1$, where still $z\notin \{0,1\}$, into the original equation yields $$f(f(z))=1-\frac{1}{f(f(f(z)))}.$$ Denoting $f(f(z))=x$, we obtain $x=1-\frac{1}{f(x)}$, i.e., $$f(x)=\frac{1}{1-x}(5)$$ This holds for any real number $x$ except 0 and 1. By the condition of the problem, $f(0)=1$, which implies the equality (5) also in the case $x=0$.

An easy check shows that $f(x)=\frac{1}{1-x}$ satisfies all conditions of the problem.

---

Alternative solution.

Plugging in $y=\frac{c}{x}$ for $c\notin \{0,1\}$ gives $f(f(c))=1-\frac{x}{cf(f(x))}$.

On the other hand, $y=1$ and $x=c$ gives $f(f(c))=1-\frac{1}{f(f(f(c)))}$, from which we conclude that $\frac{x}{f(f(f(x)))}$ is constant for all $x\ne 0,x\ne 1$. Thus $f(f(f(x)))=ax$ for some constant $a$. Now, after substituting $y=1$ into the original equation, applying $f(f(f(x)))=ax$ gives $$f(f(x))=1-\frac{1}{ax}(6)$$ Like in Solution 1, note that $f(x)\ne 0$, $f(x)\ne 1$ whenever $x\ne 0,x\ne 1$. Thus we can substitute $f(x)$ for $x$ in (6), yielding $$ax=f(f(f(x)))=1-\frac{1}{af(x)},$$ which in turn implies $f(x)=\frac{1}{a}\cdot \frac{1}{1-ax}$. Thus $$\begin{array}{rl}f(f(x))& =\frac{1}{a}\cdot \frac{1}{1-a\cdot \frac{1}{a}\cdot \frac{1}{1-ax}}=\frac{1}{a}\cdot \left(1-\frac{1}{ax}\right),\\ f(f(f(x)))& =\frac{1}{a}\cdot \left(1-\frac{1}{a\cdot \frac{1}{a}\cdot \frac{1}{1-ax}}\right)=x.\end{array}$$ Consequently, $a=1$ and $f(x)=\frac{1}{1-x}$ whenever $x\ne 0,x\ne 1$. The condition $f(0)=1$ implies that $f(x)=\frac{1}{1-x}$ holds for $x=0$, too. An easy check shows that $f(x)=\frac{1}{1-x}$ satisfies all conditions of the problem.

---

Alternative solution.

Like in Solution 1, we show that if $x\ne 0,x\ne 1$ then $f$ can be applied to $x$ infinitely many times.

We note that if $x\notin \{0,1\}$ then $f(f(f(x)))=x$. Indeed, if we suppose the contrary for any $x\notin \{0,1\}$, plugging $y=\frac{1}{f(f(f(x)))}$ into the given equation yields $f\left(f\left(\frac{x}{f(f(f(x)))}\right)\right)=0$. This violates the condition of the problem.

Hence the original equation reduces to $f(f(xy))=1-\frac{1}{xy}$ whenever $xy\ne 0,xy\ne 1,x\ne 1$. After applying $f$ to both sides of this equation, using $f(f(f(x)))=x$ yields $xy=f\left(1-\frac{1}{xy}\right)$. Denoting $z=1-\frac{1}{xy}$ gives $f(z)=\frac{1}{1-z}$ for all real numbers $z$ except 0 and 1, because all such real numbers can be represented in the form $1-\frac{1}{xy}$. The condition $f(0)=1$ implies that $f(x)=\frac{1}{1-x}$ holds for $x=0$, too. An easy check shows that $f(x)=\frac{1}{1-x}$ satisfies all conditions of the problem.