BMO 2017

Let $p$ be a prime number. Then for $x=y=p$ the given condition gives us that the number $f^2(p)+3pf(p)$ is a perfect square. Then, $f^2(p)+3pf(p)=k^2$ for some positive integer $k$. Completing the square gives us that $(2f(p)+3p{)}^2-9p^2=4k^2$, or $$(2f(p)+3p-2k)(2f(p)+3p+2k)=9p^2.(1)$$ Since $2f(p)+3p+3k>3p$, we have the following 4 cases. $$\begin{gathered} \{\begin{array}{l}2f(p)+3p+2k=9p\\ 2f(p)+3p-2k=p\end{array}\text{ or }\{\begin{array}{l}2f(p)+3p+2k=p^2\\ 2f(p)+3p-2k=9\end{array}\text{ or} \\ \{\begin{array}{l}2f(p)+3p+2k=3p^2\\ 2f(p)+3p-2k=3\end{array}\text{ or }\{\begin{array}{l}2f(p)+3p+2k=9p^2\\ 2f(p)+3p-2k=1\end{array} \end{gathered}$$ Solving the systems, we have the following cases for $f(p)$. $$f(p)=p\text{ or }f(p)={\left(\frac{p-3}{2}\right)}^2\text{ or }f(p)=\frac{3p^2-6p-3}{4}\text{ or }f(p)={\left(\frac{3p-1}{2}\right)}^2.$$ In all cases, we see that $f(p)$ can be arbitrary large whenever $p$ grows. Now fix a positive integer $x$. From the given condition we have that $$(f(y)+x{)}^2+xf(x)-x^2$$ is a perfect square. Since for $y$ being a prime, let $y=q$, $f(q)$ can be arbitrary large and $xf(x)-x^2$ is fixed, it means that $xf(x)-x^2$ should be zero, since the difference of $(f(q)+x+1{)}^2$ and $(f(q)+x{)}^2$ can be arbitrary large. After all, we conclude that $xf(x)=x^2$, so $f(x)=x$, which clearly satisfies the given condition.