jmc

We consider the different cases:

$\bullet$ $1$ by $1$ square: There are $4$ of those (formed by joining adjacent points).

$\bullet$ $2$ by $2$ square: There is $1$ of those (formed by joining the $4$ corner points).

$\bullet$ $\sqrt{2}$ by $\sqrt{2}$ square: There is $1$ of those (formed by joining the middle points on the $4$ edges, in other words, the diagonals of the $1$ by $1$ squares).

$\bullet$ $1$ by $2$ rectangle: There are $4$ of those.

So that's a total of $4+1+1+4=\boxed{10}.$