smc

Let the rectangle be $ABCD$ with $AB=CD=w$ and $AD=BC=5$, as in the diagram. We desire a line such that reflecting point $C$ across that line yields point $A$. For this to happen, the line must be perpendicular to the diagonal $\overline{AC}$, and it must go through the midpoint of $\overline{AC}$ (let it be point $M$). Let the intersection of this line with $\overline{BC}$ be point $X$ and with $\overline{AD}$ be point $Y$. From the problem, we know that $XY=\sqrt{6}$. By HL congruence, $△AMY\cong △CMX$, so $AM=CM=x$, where $x$ is some number. Furthermore, $XM=MY=\frac{\sqrt{6}}{2}$. By AA similarity, $△MXC\sim △BAC$, so $\frac{MX}{MC}=\frac{BA}{BC}$. $MX=\frac{\sqrt{6}}{2}$, $MC=x$, $BA=w$, and $BC=5$, so we can rewrite this proportion to solve for $x$ in terms of $w$: $$\begin{array}{rl}\frac{\sqrt{6}/2}{x}& =\frac{w}{5}\\ \frac{5\sqrt{6}}{2}& =xw\\ x& =\frac{5\sqrt{6}}{2w}\end{array}$$ By the Pythagorean Theorem on $△ABC$, we know that $w^2+25=4x^2$, and we can plug in our new expression for $x$ into this equation to solve for $w$: $$\begin{array}{rl}{w}^2+25& =4(\frac{5\sqrt{6}}{2w}{)}^2\\ w^2+25& =\frac{25\ast 6}{w^2}\\ w^4+25w^2-150& =0\\ (w^2-5)(w^2+30)& =0\end{array}$$ Because $w>0$, $w^2=5$, and so $w=\boxed{\sqrt{5}}$, which corresponds to answer choice $\boxed{\sqrt{5}}$.