75th Romanian Mathematical Olympiad

If one of the matrices $A$, $B$, $C$ is null, then all the matrices are null. Suppose that there are three matrices $A$, $B$, $C\in {\mathcal{M}}_2(\mathbb{R})\setminus \{O_2\}$ which satisfy the equations from the statement. We have $\text{tr}(A)=\text{tr}(BC-CB)=0$. Similarly, $\text{tr}(B)=\text{tr}(C)=0$. From the equation $A^2-\text{tr}(A)A+\det (A)I_2=O_2$, we obtain $A^2=a{I}_2$, where $a=-\det (A)$. Similarly, we have $B^2=b{I}_2$ and $C^2=c{I}_2$, with $b=-\det (B)$ and $c=-\det (C)$. Multiplying the equation $A=BC-CB$ on the left and on the right by $B$, we obtain the relations $BA=B^{2}C-BCB=bC-BCB$ and $AB=BCB-C{B}^2=BCB-bC$, respectively. Then $AB+BA=O_2$. Thus, the equation $C=AB-BA$ leads to $C=2AB$. Similarly, $A=2BC$ and $B=2CA$. Then $A=-2CB=-2C(2CA)=-4C^{2}A=-4cA$. Similarly, $B=-4aB$ and $C=-4bC$.

Since the matrices $A$, $B$ and $C$ are assumed to be non-null, we obtain $a=b=c=-1/4$. So, we have $A=$ and $B=$, with $x,y,z,s,t,u\in \mathbb{R}$, such that $x^2+yz=-1/4$ and $s^2+tu=-1/4$. From the relation $AB+BA=O_2$, we find $2xs=-(yu+zt)$. Thus, $4x^2{s}^2=(yu+zt{)}^2=(yu-zt{)}^2+4yztu\ge 4(yz)(tu)=4\left(x^2+\frac{1}{4}\right)\left(s^2+\frac{1}{4}\right)$. But $x^2{s}^2<\left(x^2+\frac{1}{4}\right)\left(s^2+\frac{1}{4}\right)$, for all $x,s\in \mathbb{R}$. Contradiction.

Therefore, the unique solution of the given system is $A=B=C=O_2$.

---

Alternative solution.

Let $A$, $B$, $C\in M_2(\mathbb{R})$ be three matrices that satisfy the given system. We have $\text{tr}(A)=\text{tr}(BC-CB)=0$. We also deduce $\text{tr}(B)=\text{tr}(C)=0$. Then $A=$, $B=$, $C=$, $a_i,b_i,c_i\in \mathbb{R}$, for $i=\overline{1,3}$. We obtain $BC-CB=$. Hence, we have $a_2=2(b_1{c}_2-b_2{c}_1)$ and $a_2^2=2a_2(b_1{c}_2-b_2{c}_1)=2a_2{b}_1{c}_2-2a_2{b}_2{c}_1$. Similarly, we obtain $b_2^2=2b_2{c}_1{a}_2-2b_2{c}_2{a}_1$ and $c_2^2=2c_2{a}_1{b}_2-2c_2{a}_2{b}_1$. Then $a_2^2+b_2^2+c_2^2=0$. Since $a_2,b_2,c_2\in \mathbb{R}$, we conclude $a_2=b_2=c_2=0$. Analogously, we can show that $a_3=b_3=c_3=0$. We obtain $a_1=b_2{c}_3-b_3{c}_2=0$. Similarly, $b_1=c_1=0$.

In conclusion, the unique solution of the given system is $A=B=C=O_2$.