Belarusian Mathematical Olympiad

Answer: $a=b=2$. Let $A(\alpha ,0)$, $B(\beta ,0)$, $C(0,\gamma )$, and $M(a,b)$. Since $M$ is the center of the circumscribed circle of the triangle $ABC$, $M$ lies on the perpendicular bisector of the segment $AB$. Therefore, the projection of $M$ on the $x$-axis is the midpoint of the segment $AB$, i.e. $a=(\alpha +\beta )/2$. Since the points $A$ and $B$ lie on the parabola, their abscissae satisfy the equation $x^2-2bx+a+1=0$, then, by Vieta's theorem, $a=(\alpha +\beta )/2=2b/2=b$. Substituting $x=0$ into the parabola equation, we find the ordinate $\gamma$ of $C$: $\gamma =a+1$. Let $R$ be the radius of the circle $\Gamma$. Since $C$ lies on $\Gamma$, we obtain $$R^2=|MC{|}^2=(a-0{)}^2+(b-\gamma {)}^2=[a=b,\gamma =a+1]=a^2+1.$$ Since $A$ and $B$ lie on $\Gamma$, we get $$a^2+1=R^2=|MA{|}^2=|MB{|}^2\Rightarrow (a-\alpha {)}^2+(b-0{)}^2=$$ $$=(a-\beta {)}^2+(b-0{)}^2=(a-\alpha {)}^2+a^2=(a-\beta {)}^2+a^2\Rightarrow (a-\alpha {)}^2=(a-\beta {)}^2=1.$$ Without loss of generality we can assume that $\beta >\alpha$, then $\alpha =a-1$, $\beta =a+1$. The numbers $\alpha =a-1$ and $\beta =a+1$ are the roots of the equation $x^2-2bx+a+1=0$, so, by Vieta's theorem $$a^2-1=(a-1)(a+1)=\alpha \beta =a+1\Rightarrow a^2-a-2=0.$$ We see that $a=-1$ and $a=2$ are the roots of this equation.

For $a=-1$ we have $\gamma =0$, but then $C(0,0)$, i.e. $C$ coincides with the origin, contrary to the problem condition. For $a=2$ we have $M(2,2)$, the parabola equation has the form $y=x^2-4x+3$, and the circle equation has the form $(x-2{)}^2+(y-2{)}^2=5$. It is easy to verify that all problem conditions hold in this case.