Hong Kong Preliminary Selection Contest

Since $a+b$ satisfies the given equation, we have $(a+b{)}^2+a(a+b)+b=0$. Rearranging gives $b^2+(3a+1)b+2a^2=0$. This means $b$ is a root of the equation $x^2+(3a+1)x+2a^2=0$. As $a$ and $b$ are integers, the discriminant $(3a+1{)}^2-4\cdot 2a^2=a^2+6a+1$ must be a perfect square. Let $a^2+6a+1=m^2$. Then $(a+3{)}^2-8=m^2$, and so $(a+3-m)(a+3+m)=8$. As the two terms $a+3-m$ and $a+3+m$ have the same parity (they differ by $2m$ which is even), they can only be $2$ and $4$, or $-2$ and $-4$ (up to permutation). The corresponding possible values of $a$ are $0$ and $-6$. When $a=0$, the equation becomes $b^2+b=0$, giving $b=0$ or $b=-1$. When $a=-6$, the equation becomes $b^2-17b+72=0$, giving $b=8$ or $b=9$. The smallest possible value of $ab$ is thus $(-6)\times 9=-54$.