69th Belarusian Mathematical Olympiad

Answer: $P(x)=a{x}^2-a(b+1)x+ab$, $Q(x)=x^2-(b+1)x+2b$, where $a,b\in \mathbb{R}$ and $a\ne 0$.

Denote the degrees of polynomials $P$ and $Q$ by $m$ and $n$, respectively. In the equality $$P(Q(x))=P(x)Q(x)-P(x)(1)$$ compare the degrees of both sides: $$mn=m+n⟺(m-1)(n-1)=1⟺m=n=2.$$ Therefore we can denote $P(x)=\alpha x^2+\beta x+c$. Hence (1) can be written in the form $\alpha (Q(x){)}^2+\beta Q(x)=P(x)Q(x)-(P(x)+c)Q(x)$. Since polynomials $P$ and $Q$ have equal degrees, $P(x)+c=aQ(x)$ for some real number $a\ne 0$. The equality (1) now takes the form $$aQ(Q(x))-c=(aQ(x)-c)Q(x)-(aQ(x)-c)=a(Q(x){)}^2-(a+c)Q(x)+c.$$ Since $Q$ is nonconstant, it attends infinitely many values, hence $aQ(x)=a{x}^2-(a+c)x+2c$, or equivalently $Q(x)=x^2-(b+1)x+2b$ (where $b=\frac{c}{a}$). Then $P(x)=aQ(x)-ab$, i.e. $P(x)=a{x}^2-a(b+1)x+ab$. It is easy to verify that such polynomials $P$ and $Q$ satisfy the problem conditions (1). Indeed, since $P$ and $Q$ satisfy $P(x)=aQ(x)-ab$, the equality (1) can be written as $aQ(Q(x))-ab=(aQ(x)-ab)Q(x)-(aQ(x)-ab)$, or, after simplification, $Q(Q(x))=(Q(x){)}^2-(b+1)Q(x)+2b$, which is true for $Q(x)=x^2-(b+1)x+2b$.