THE 68th ROMANIAN MATHEMATICAL OLYMPIAD

We will prove that the result is valid for $n\le 3$ and not true for $n\ge 4$. Matrices $C=AB$ and $D=BA$ have the same trace and both determinants are zero. If $C^2=O_n$, then $D^3=B{C}^{2}A=O_n$.

For $n=2$, matrices $C$ and $D$ have the same characteristic polynomial $P_C(X)=P_D(X)=X^2-aX$, with $a=\text{tr}C=\text{tr}D$. By the hypothesis and the Cayley-Hamilton we get $O_n=C^3=a{C}^2$. Thus $a=0$ or $C^2=0$. Both cases imply $D^2=aD=O_n$

We may now suppose $C^2\ne O_n$. For $n=3$, the characteristic polynomials are $P_C(X)=X^3-a{X}^2+bX$ and $P_D(X)=X^3-a{X}^2+cX$ respectively. From $a{C}^2=bC$ we get $b{C}^2=O_n$, $b=0$. So $a{C}^2=O_n$, which implies $a=0$.

Applying again Hamilton-Cayley we obtain $D^4=-c{D}^2$. As $D^4=B{C}^{3}A=O_n$, we obtain $c{D}^2=O_n$, so that we must have $D^2=O_n$, or $c=0$. Both imply $D^3=O_n$

To deal with the case $n=4$, consider $$A=\left(\begin{array}{cccc}1& 0& 0& 0\\ 0& 0& 0& 1\\ 0& 0& 1& 0\\ 0& 0& 1& 0\end{array}\right),B=\left(\begin{array}{cccc}0& 1& 0& 0\\ 0& 0& 0& 1\\ 1& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right),$$ matrices that show that we can have $(AB{)}^3=O_n$ and $(BA{)}^3\ne O_n$.

For $n\ge 5$ consider matrices which have the upper left corner the 4 by 4 matrices in the previous examples and zeroes in rest.