jmc

Let $s$ be the side length of the square. Also let $D$ be the vertex of the square on side $AC$, and let $E$ be the vertex of the square on side $AB$. Let $F$ and $G$ be the feet of the altitudes from $D$ and $A$ to $BC$, respectively. Let $x$ be the length of $AD$.



Without loss of generality, we assume that $AC>AB$, as in the diagram. From the given information, we know that $AC=8$, $BC=10$, and $DC=8-x$. We can find that $AG=AB\cdot AC/BC=4.8$.

From similar triangles $AED$ and $ABC$, we find that $s/10=x/8$. From similar triangles $DFC$ and $AGC$, we have $s/4.8=(8-x)/8$. Summing these two equations, we have $$\frac{s}{10}+\frac{s}{4.8}=\frac{x}{8}+\frac{8-x}{8}$$$$\frac{14.8s}{48}=1.$$Solving for $s$, we find that $s=\boxed{\frac{120}{37}}$.