66th Czech and Slovak Mathematical Olympiad

We distinguish several cases.

First, assume $x=y=z$. Then the whole system reduces to $(k+2)x^2=x$. Its solution is a triplet $(0,0,0)$ for any $k$ and moreover triplet $\left(\frac{1}{k+2},\frac{1}{k+2},\frac{1}{k+2}\right)$ if $k\ne -2$.

Let's get back to the original system. Subtracting the second equation from the first one yields $$(x^2-z^2)+ky(x-z)=z-x$$ which rewrites as $$(x-z)(x+z+ky+1)=0.\text{\hspace{1em}(1)}$$ Similarly, subtracting the third equation from the second one yields $$(y-x)(y+x+kz+1)=0.\text{\hspace{1em}(2)}$$ If $x\ne y\ne z\ne x$, the equations (1), (2) reduce to $$\begin{array}{rl}x+z+ky+1& =0,\\ y+x+kz+1& =0.\end{array}$$ Subtracting these two equations we arrive at $(y-z)(k-1)=0$ implying that $k=1$ and $x+y+z=-1$. However that's impossible since for $k=1$ we get $$z=x^2+xy+y^2={\left(x+\frac{y}{2}\right)}^2+\frac{3y^2}{4}\ge 0$$ and likewise $x\ge 0$ and $y\ge 0$ so altogether $x+y+z\ge 0$.

We found out that in every solution to the original system, some two unknowns have the same value. As the system is cyclic, let us from now on assume $x=y\ne z$ (the case $x=y=z$ has already been solved). Equation (1) then implies $x+y+ky+1=0$, that is $x=-(k+1)y-1$, and the original system reduces to a single equation $$(k+2)y^2+(k+1)y+1=0.\text{\hspace{1em}(3)}$$ Let us remark that any solution to equation (3) is a solution we haven't found yet, because equality $x=y$ i.e. $y=-(k+1)y-1$ is only possible for $k\ne -2$ and yields $x=y=z=-1/(k+2)$ which is not a solution to the original system.

For $k=-2$ the equation (3) is linear with a unique solution $y=1$. This yields solution $(0,1,1)$ and its two permutations.

For $k\ne -2$ the equation (3) is quadratic and has real solutions if and only if $$D=(k+1{)}^2-4(k+2)=k^2-2k-7\ge 0,$$ which translates to $k\notin (1-2\sqrt{2},1+2\sqrt{2})$. For $k=1\pm 2\sqrt{2}$ there is a unique solution $$y_0=-\frac{k+1}{2(k+2)}=1\mp \sqrt{2}\text{a}{x}_0=\frac{(k+1{)}^2}{2(k+2)}-1=1.$$ which yields the three permutations of $(x_0,y_0,y_0)$ as solutions to the original system.

For $k\in (-\infty ,-2)\cup (-2,1-2\sqrt{2})\cup (1+2\sqrt{2},\infty )$, the quadratic equation (3) has two distinct solutions $$y_{1,2}=\frac{-k-1\pm \sqrt{k^2-2k-7}}{2(k+2)},$$ that give two distinct values $x_{1,2}=-(k+1)y_{1,2}-1$. The original system thus has six solutions: three permutations of $(x_1,y_1,y_1)$ and three permutations of $(x_2,y_2,y_2)$.

The following table summarizes the number of solutions to the given system in terms of $k$:

Interval for $k$	$(0,0,0)$	$(1/(k+2),1/(k+2),1/(k+2))$	Equation (3)	Total
$(-\infty ,-2)$	1	1	6	8
$-2$	1	0	3	4
$(-2,1-2\sqrt{2})$	1	1	6	8
$1-2\sqrt{2}$	1	1	3	5
$(1-2\sqrt{2},1+2\sqrt{2})$	1	1	0	2
$1+2\sqrt{2}$	1	1	3	5
$(1+2\sqrt{2},\infty )$	1	1	6	8