jmc

Note that the axis of symmetry of the parabola is $x=\frac{-(-8)}{2\cdot 1}=4.$

Let $2t$ be the side length of the square. Then $$\begin{array}{rl}A& =(4-t,0),\\ B& =(4+t,0),\\ C& =(4+t,-2t),\\ D& =(4-t,-2t).\end{array}$$But $C$ lies on the parabola $y=x^2-8x+12=(x-4{)}^2-4,$ so $$-2t=t^2-4.$$Then $t^2+2t-4=0,$ so by the quadratic formula, $$t=-1\pm \sqrt{5}.$$Since $t$ is half a side length, it must be positive, and so $t=-1+\sqrt{5}.$ Therefore, the area of the square is $$(2t{)}^2=(-2+2\sqrt{5}{)}^2=\boxed{24-8\sqrt{5}}.$$