AustriaMO2013

The square divides the circle into four arcs. None of them can contain more than one of the triangle vertices, because the distance between two triangle vertices is equivalent to an inner angle of $120^{\circ }$, whereas the distance between two vertices of the square is only equivalent to an inner angle of $90^{\circ }$. Therefore, the triangle vertices must be on three distinct parts of the circle. Let $ABCD$ denote the square and let $PQR$ denote the triangle. Without loss of generality, let us assume that $P$ is on the arc between $A$ and $B$, that $Q$ is between $B$ and $C$, and that $R$ is between $D$ and $A$, as shown in the graph below.



We see that the heptagon is combined of the square $ABCD$ and the three triangles $APB$, $BQC$ and $DRA$. Since the size of the square $ABCD$ is constant, it is sufficient to maximize resp. minimize the sum of the areas of these three triangles. Let $h_1$ denote the distance between point $P$ and line $AB$ (i.e., the height of triangle $APB$), let $h_2$ denote the distance between point $Q$ and line $BC$, and let $h_3$ denote the distance between point $R$ and line $DA$. The sum of the areas of the three triangles can thus be calculated as $\frac{|AB|}{2}\cdot h_1+\frac{|BC|}{2}\cdot h_2+\frac{|DA|}{2}\cdot h_3=\frac{s}{2}\cdot (h_1+h_2+h_3)$, where $s$ denotes the length of each side of the square. Since $s$ is constant, it is therefore sufficient to maximize resp. minimize the sum $(h_1+h_2+h_3)$.

We will do this by separately maximizing resp. minimizing the height $h_1$, and the sum of the heights $h_2+h_3$.

The height $h_1$ is largest when $P$ is exactly in the middle of the arc between $A$ and $B$.

For maximizing the sum $h_2+h_3$, consider the rectangle $QXRY$ with sides parallel to the sides of the square $ABCD$ and with $QR$ as one of its diagonals. By Pythagoras it holds that $|QR{|}^2=|QX{|}^2+|XR{|}^2=(h_1+s+h_2{)}^2+|XR{|}^2$, and therefore $(h_2+s+h_3{)}^2=|QR{|}^2-|XR{|}^2$. Since the length of the triangle side $QR$ is constant, the expression (and consequently the sum $h_2+h_3$) is largest when $|XR|=0$. This is the case if side $QR$ is parallel to side $CD$, or equivalently, if $P$ is exactly in the middle of the arc between $A$ and $B$.

Since $h_1$ and the sum $h_2+h_3$ are both maximized in the same case, the sum of all three is also largest when $P$ is exactly in the middle of the arc between $A$ and $B$.

For determining the minimum, we again separately look at $h_1$ and the sum $h_2+h_3$ and minimize them under the condition that the triangle vertices must remain on the correct parts of the circle.

The height $h_1$ becomes smaller the closer $P$ moves towards either $A$ or $B$. Since $Q$ must remain between $B$ and $C$, and $R$ must remain between $D$ and $A$, the minimum is reached if either $Q=C$ or $R=D$.

Likewise, the sum $h_2+h_3$ becomes smaller if $|XR|$ becomes larger, so again the minimum is reached if $Q=C$ or $R=D$. $\square$