jmc

The largest possible square is the square with one vertex on the triangles' coincident vertices and with sides parallel to and coincident with those of the big square. There are two of them. We draw them in and label the diagram as shown: First, we find the side length of the equilateral triangle. $M$ is the midpoint of $EF$; let $MF=x$, so $AM=MC=x\sqrt{3}$ and $AC=2x\sqrt{3}$. $AC$ is the diagonal of $ABCD$ and thus has length $10\sqrt{2}$. So we have $$2x\sqrt{3}=10\sqrt{2}.$$ It follows that the side length of the triangle is $2x=\frac{10\sqrt{2}}{\sqrt{3}}$.

Now, look at diagonal $BD$ and notice that it is made up of twice the diagonal of the small square plus the side length of the triangle. Let the side length of the small square be $y$, so we have $$BD=BE+EF+FD=y\sqrt{2}+\frac{10\sqrt{2}}{\sqrt{3}}+y\sqrt{2}=10\sqrt{2}.$$ Solving yields $y\sqrt{2}=5\sqrt{2}-\frac{5\sqrt{2}}{\sqrt{3}}$ so $y=\boxed{5-\frac{5\sqrt{3}}{3}}$.