75th Romanian Mathematical Olympiad

a) Define $C=2A-I_n$ and $D=2B-I_n$. We have $CD-I_n=-(DC-I_n)=2(AB-BA)$. As $n$ is odd, $\det (CD-I_n)=-\det (DC-I_n)$. As for any two square matrices $X,Y$, we have $\det (XY-I_n)=\det (YX-I_n)$, in our case we get $\det (CD-I_n)=-\det (CD-I_n)$, that is $\det (CD-I_n)=0$. Consequently $\det (AB-BA)=0$.

b) Define $E=AB-BA$. Suppose $\det (E)\ne 0$. We have $AE+EA=A^{2}B-B{A}^2=A(A+B-BA)-(A+B-AB)A=AB-BA=E$. As $E$ is invertible, $E^{-1}AE+A=I_n$. Using the property $\mathrm{tr}(E^{-1}AE)=\mathrm{tr}(A)$, we get $\mathrm{tr}(A)=\frac{n}{2}$. By symmetry, $\mathrm{tr}(B)=\frac{n}{2}$, implying $\mathrm{tr}(A)=\mathrm{tr}(B)$, in contradiction with the condition b). That is $\det (AB-BA)=0$.