Baltic Way 2023 Shortlist

Plugging in $y=\frac{a}{x}$ for $a,x\in D$ gives $$f(f(a))=1-\frac{x}{af(f(f(x)))}.$$ On the other hand, $y=1$ and $x=a$ gives for $a\in D$ $$f(f(a))=1-\frac{1}{f(f(f(a)))}.$$ from which we conclude $\frac{x}{f(f(f(x)))}$ is constant for all $x\in D$. Thus it follows $f(f(f(x)))=Cx$ for some constant $C\ne 0$. Plugging this into the second equation gives $f(f(x))=1-\frac{1}{Cx}$. Replace here $x$ with $f(x)$. This is allowed, since $f(x)\in D$ follows from the definition of $f$. Thus $$Cx=f(f(f(x)))=1-\frac{1}{Cf(x)},\text{ i.e. }f(x)=\frac{1}{C(1-Cx)}.$$

So we get $f(x)=\frac{1}{1-x}$ for all $x\in D$, which clearly solves the given functional equation, so we are done.