jmc

If $a$ is not relatively prime with $24$, then the modular inverse of $a$ does not exist. Multiplying both sides of the congruence by $a$ yields that $a^2\equiv 1(\mathrm{mod}24)$, or equivalently that $a^2-1\equiv (a+1)(a-1)\equiv 0(\mathrm{mod}24)$. Since $a$ is not divisible by $3$, it follows that at least one of $a+1$ or $a-1$ must be divisible by $3$. Also, since $a$ is not divisible by $2$, then both $a+1$ and $a-1$ are even, and exactly one of them is divisible by $4$. Thus, $3\times 2\times 4=24$ will always divide into $(a+1)(a-1)$, and so the statement is true for every integer $a$ relatively prime to $24$. The answer is the set of numbers relatively prime to $24$, namely $\{1,5,7,11,13,17,19,23\}$. There are $\boxed{8}$ such numbers.

The number of positive integers smaller than and relatively prime to $24$ is also given by the Euler's totient function.