jmc

The two vertices that lie on $y=x^2$ must lie on a line of the form $y=2x+k.$ Setting $y=x^2,$ we get $x^2=2x+k,$ so $x^2-2x-k=0.$ Let $x_1$ and $x_2$ be the roots of this quadratic, so by Vieta's formulas, $x_1+x_2=2$ and $x_1{x}_2=-k.$

The two vertices on the parabola are then $(x_1,2x_1+k)$ and $(x_2,2x_2+k),$ and the square of the distance between them is $$\begin{array}{rl}(x_1-x_2{)}^2+(2x_1-2x_2{)}^2& =5(x_1-x_2{)}^2\\ & =5[(x_1+x_2{)}^2-4x_1{x}_2]\\ & =5(4+4k)\\ & =20(k+1).\end{array}$$

The point $(0,k)$ lies on the line $y=2x+k,$ and its distance to the line $y-2x+17=0$ is $$\frac{|k+17|}{\sqrt{5}}.$$Hence, $$20(k+1)=\frac{(k+17{)}^2}{5}.$$This simplifies to $k^2-66k+189=0,$ which factors as $(k-3)(k-63)=0.$ Hence, $k=3$ or $k=63.$

We want to find the smallest possible area of the square, so we take $k=3.$ This gives us $20(k+1)=\boxed{80}.$