75th Romanian Mathematical Olympiad

If $b=0$, then $BA=a{I}_n$, so $A$ and $B$ are invertible. We will suppose $b\ne 0$.

Denote by $\sigma (X)$ the set of eigenvalues of a matrix $X\in {\mathcal{M}}_n(\mathbb{C})$. Let $\lambda \in \sigma (AB)$. Then $$\det (BA-(b\lambda +a)I_n)=\det (bAB-b\lambda I_n)=b^n\det (AB-\lambda I_n)=0,$$ implying $b\lambda +a\in \sigma (BA)$. As $\sigma (AB)=\sigma (BA)$, if $\lambda \in \sigma (AB)$, we get $b\lambda +a\in \sigma (AB)$.

Define $f:\mathbb{C}\to \mathbb{C}$, $f(z)=bz+a$, for all $z\in \mathbb{C}$, and for $k\in {\mathbb{N}}^{\ast }$, denote by $f^{[k]}=\underset{k\text{ times}}{\underbrace{f\circ f\circ f\cdots \circ f}}$. We conclude inductively, that if $\lambda \in \sigma (AB)$, then $f^{[k]}(\lambda )\in \sigma (AB)$, for $k\in {\mathbb{N}}^{\ast }$.

Suppose $0\in \sigma (AB)$. Then $f(0),f^{[2]}(0),\dots ,f^{[n+1]}(0)\in \sigma (AB)$. As $\sigma (AB)$ has at most $n$ elements, there are $p,q\in \{1,2,\dots ,n+1\}$, $p<q$, such that $f^{[p]}(0)=f^{[q]}(0)$.

Because $f^{[k]}(z)=b^{k}z+a\frac{b^k-1}{b-1}$, $z\in \mathbb{C}$, $k\in {\mathbb{N}}^{\ast }$, we obtain $a\frac{b^p-1}{b-1}=a\frac{b^q-1}{b-1}$, that is $b^{q-p}=1$, a contradiction.

In conclusion, $0\notin \sigma (AB)$, so $\det (A)\cdot \det (B)=\det (AB)\ne 0$, and the result follows.