imc

We can take half of a cross section of the sphere, as such: Notice that we chose a cross section where one of the spheres was tangent to the lateral surface of the cone at $D$. To evaluate $r$, we will find $AE$ and $EC$ in terms of $r$; we also know that $AE+EC=5$, so with this, we can solve $r$. Firstly, to find $EC$, we can take a bird's eye view of the cone: Note that $C$ is the centroid of equilateral triangle $EXY$. Also, since all of the medians of an equilateral triangle are also altitudes, we want to find two-thirds of the altitude from $E$ to $XY$; this is because medians cut each other into a $2$ to $1$ ratio. This equilateral triangle has a side length of $2r$, therefore it has an altitude of length $r\sqrt{3}$; two thirds of this is $\frac{2r\sqrt{3}}{3}$, so $EC=\frac{2r\sqrt{3}}{3}.$ To evaluate $AE$ in terms of $r$, we will extend $\overline{OE}$ past point $O$ to $\overline{AB}$ at point $F$.$△AEF$ is similar to $△ACB$. Also, $AO$ is the angle bisector of $\angle EAB$. Therefore, by the angle bisector theorem, $\frac{OE}{OF}=\frac{AE}{AF}=\frac{5}{13}$. Also, $OE=r$, so $\frac{r}{OF}=\frac{5}{13}$, so $OF=\frac{13r}{5}$. This means that$$AE=\frac{5\cdot EF}{12}=\frac{5\cdot (OE+OF)}{12}=\frac{5\cdot (r+\frac{13r}{5})}{12}=\frac{18r}{12}=\frac{3r}{2}.$$ We have that $EC=\frac{2r\sqrt{3}}{3}$ and that $AE=\frac{3r}{2}$, so $AC=EC+AE=\frac{2r\sqrt{3}}{3}+\frac{3r}{2}=\frac{4r\sqrt{3}+9r}{6}$. We also were given that $AC=5$. Therefore, we have $$\frac{4r\sqrt{3}+9r}{6}=5.$$ This is a simple linear equation in terms of $r$. We can solve for $r$ to get $r=\boxed{\frac{90-40\sqrt{3}}{11}}.$