51st IMO Shortlisted Problems

By substituting $y=1$, we get $$f\left(f(x{)}^2\right)=x^{3}f(x)$$ Then, whenever $f(x)=f(y)$, we have $$x^3=\frac{f\left(f(x{)}^2\right)}{f(x)}=\frac{f\left(f(y{)}^2\right)}{f(y)}=y^3$$ which implies $x=y$, so the function $f$ is injective.

Now replace $x$ by $xy$ in the previous equation, and apply the original equation twice, the second time to $(y,f(x{)}^2)$ instead of $(x,y)$: $$f\left(f(xy{)}^2\right)=(xy{)}^{3}f(xy)=y^{3}f\left(f(x{)}^{2}y\right)=f\left(f(x{)}^{2}f(y{)}^2\right)$$ Since $f$ is injective, we get $$\begin{array}{rl}f(xy{)}^2& =f(x{)}^{2}f(y{)}^2\\ f(xy)& =f(x)f(y)\end{array}$$ Therefore, $f$ is multiplicative. This also implies $f(1)=1$ and $f\left(x^n\right)=f(x{)}^n$ for all integers $n$.

Then the functional equation can be re-written as $$\begin{array}{rl}f(f(x){)}^{2}f(y)& =x^{3}f(x)f(y),\\ f(f(x))& =\sqrt{x^{3}f(x)}\end{array}$$ Let $g(x)=xf(x)$. Then, we have $$\begin{array}{rl}g(g(x))& =g(xf(x))=xf(x)\cdot f(xf(x))=xf(x{)}^{2}f(f(x))=\\ & =xf(x{)}^2\sqrt{x^{3}f(x)}=(xf(x){)}^{5/2}=(g(x){)}^{5/2}\end{array}$$ and, by induction, $$\underset{n+1}{\underbrace{g(g(\dots g}}(x)\dots ))=(g(x){)}^{(5/2{)}^n}$$ for every positive integer $n$.

Consider this for a fixed $x$. The left-hand side is always rational, so $(g(x){)}^{(5/2{)}^n}$ must be rational for every $n$. We show that this is possible only if $g(x)=1$. Suppose that $g(x)\ne 1$, and let the prime factorization of $g(x)$ be $g(x)=p_1^{{\alpha }_1}\dots p_k^{{\alpha }_k}$ where $p_1,\dots ,p_k$ are distinct primes and ${\alpha }_1,\dots ,{\alpha }_k$ are nonzero integers. Then the unique prime factorization is $$\underset{n+1}{\underbrace{g(g(\dots g}}(x)\dots ))=(g(x){)}^{(5/2{)}^n}=p_1^{(5/2{)}^n{\alpha }_1}\dots p_k^{(5/2{)}^n{\alpha }_k}$$ where the exponents should be integers. But this is not true for large values of $n$, for example ${\left(\frac{5}{2}\right)}^n{\alpha }_1$ cannot be an integer when $2^n\nmid {\alpha }_1$. Therefore, $g(x)\ne 1$ is impossible.

Hence, $g(x)=1$ and thus $f(x)=\frac{1}{x}$ for all $x$.

The function $f(x)=\frac{1}{x}$ satisfies the equation: $$f\left(f(x{)}^{2}y\right)=\frac{1}{f(x{)}^{2}y}=\frac{1}{{\left(\frac{1}{x}\right)}^{2}y}=\frac{x^3}{xy}=x^{3}f(xy)$$