50th Mathematical Olympiad in Ukraine, Fourth Round (March 23, 2010)

Let us suppose that we can find such numbers $a,b$, that $p=a+b+2\sqrt{ab+1}$ is prime. Then $ab+1$ is complete square, $a$ and $b$ are distinct. By Cauchy-Schwartz inequality $a+b\ge 2\sqrt{ab}$. Besides this, we have $2\sqrt{ab}+1>2\sqrt{ab+1}\Rightarrow a+b\ge 2\sqrt{ab}+1$. $p$ is an odd number, because it is greater than $2$, thus $a+b-2\sqrt{ab}+1$ is natural. From the equality $$(a+b+2\sqrt{ab}+1)(a+b-2\sqrt{ab}+1)=(a-b+2)(a-b-2)\ne 0$$ it follows, that either $a-b-2$ or $a-b+2$ is divisible by $p$, but absolute value of each of these two numbers does not exceed $a+b+2$, which is less than $p$. This contradiction finishes the proof.

---

Alternative solution.

From the condition of the problem, it follows that $2\sqrt{ab+1}$ is a natural number, which immediately gives us that $z=\sqrt{ab+1}$ is also a natural number. Therefore, $b=\frac{z^2-1}{a}=\frac{(z-1)(z+1)}{a}$. This implies that $a$ can be factored $a=a_1{a}_2$ in such a way that $z-1$ is divisible by $a_1$, and $z+1$ is divisible by $a_2$.

$m=a+\frac{(z-1)(z+1)}{a}+2z=\frac{(a+z{)}^2-1}{a}=\frac{(a+z-1)(a+z+1)}{a}=\frac{a+z-1}{a_1}+\frac{a+z+1}{a_2}$. Both factors $\frac{a+z-1}{a_1}$ and $\frac{a+z+1}{a_2}$ are natural and greater than one, thus, $m$ is composite.