The Problems of Ukrainian Authors

(Edward Turkevich) Answer: the equation is true when $a=b,x=y=1-a,k=1$.

From $a+kx\le 1$: $x\le \frac{1-a}{k}$ and from $b+\frac{x}{k}\le 1$: $x\le k(1-b)$. So, $x^2\le (1-a)(1-b)$ and $x\le \frac{1}{2}(2-a-b)$ (based on the inequality of arithmetic and geometric means of $1-a$ and $1-b$). Similarly, $y^2\le (1-a)(1-b)$ and $y\le \frac{1}{2}(2-a-b)$.

$$\begin{array}{rl}(a+x{)}^2+(b+y{)}^2& =a^2+2ax+x^2+b^2+2by+y^2\\ & \le a^2+2a\cdot \frac{1}{2}(2-a-b)+(1-a)(1-b)+b^2+2b\cdot \frac{1}{2}(2-a-b)+(1-a)(1-b)\\ & =a^2+b^2+(a+b)(2-a-b)+2(1-a)(1-b)=2,\end{array}$$ What was to be proven.

The equality is true when $1-a=1-b$, so $a=b,x=y=1-a,k=1$.