Singapore Mathematical Olympiad (SMO)

Suppose $k,a,b$ satisfy the equation. Rewrite the equation as $$a=\sqrt{(k^2-4)b^2-4}=\sqrt{(kb{)}^2-4(b^2+1)}(1)$$ Consider the quadratic equation $$x^2-kbx+(b^2+1)=0(2)$$ Its solutions are $$c=\frac{kb\pm \sqrt{(kb{)}^2-4(b^2+1)}}{2}=\frac{kb\pm a}{2}(3)$$ where $a>0$, $k\ge 3$. Since $a$ and $kb$ have the same parity, $c\in {\mathbb{Z}}^{+}$. If one of the roots is $b$, then (2) becomes $(2-k)b^2+1=0$ which gives $k=3$, $b=1$ and hence $a=1$.

Let the 2 roots of (2) be $c_1,c_2$ with $c_2<c_1$. Now assume that $b\ne c_1,c_2$. Since the product of the roots is $b^2+1$, we have $c_2<b<c_1$. Now replace $b$ in (2) by $c_2$ to get $x^2-k{c}_{2}x+(c_2^2+1)=0$. If one of the roots of the equation is $c_2$, we have $k=3$. If not we can repeat the argument to get another equation with $c_2$ replaced by another positive integer $c_3<c_2$. Eventually, we must reach that case where one of the roots is equal to $c_i$.