Austrian Mathematical Olympiad

The smallest constant is $C=-1$. Equality holds for $X=Y=\frac{1}{\sqrt{3}}$ or $X=Y=-\frac{1}{\sqrt{3}}$.

We first investigate the case $X=Y$. It is easily seen that the inequality becomes equivalent to $$(3X^2-1{)}^2+4(C+1)X^2\ge 0$$ which implies $C\ge -1$ by setting $X^2=\frac{1}{3}$. It remains to prove that the inequality is true for all $X$ and $Y$ for $C=-1$. Since we had the term $(3X^2-1{)}^2$ in the above case, we compare the term $(X^2+XY+Y^2-1{)}^2$ with the terms in the given inequality and get the equivalent inequality $$(X^2+XY+Y^2-1{)}^2+(X-Y{)}^2\ge 0.$$ This is obviously true and gives the conditions $X=Y$ and $3X^2-1=0$ for equality which are the two cases $X=Y=\frac{1}{\sqrt{3}}$ and $X=Y=-\frac{1}{\sqrt{3}}$.