Belorusija 2012

Answer: $30\sqrt{3}$.

It is easy to show that if the area of the quadrilateral with three given sides is a maximum, then the quadrilateral is convex. Let $AB$, $BC$, $CD$ be given and the quadrilateral $ABCD$ have the maximal area. We have $S(ABCD)=S(ABD)+S(BCD)$ and $S(ABD)=0.5\cdot AB\cdot BD\sin \angle ABD$. If $\angle ABD\ne 90^{\circ }$, then there exists a quadrilateral with given sides having the greatest possible area. Hence $AB\perp BD$. In the same manner, we obtain $CD\perp AC$. So the right angle $ABD$ and $ACD$ subtend the segment $AD$, so the quadrilateral $ABCD$ is inscribed in the circle with the diameter $AC$.

Let $\angle CAD=\beta$, $\angle BDA=\alpha$, $AB=a$, $BC=b$, $CD=c$, $AC=x$, $BD=y$, $AD=z$. Then $$a=z\sin \alpha ,c=z\sin \beta ,x=z\cos \beta ,y=z\cos \alpha ,$$ $$\begin{array}{rl}b& =z\sin \angle CDA=z\sin (90^{\circ }-\beta -\alpha )=z\cos (\beta +\alpha )=\\ & =z(\cos \beta \cos \alpha -\sin \beta \sin \alpha ).\end{array}$$

It is easy to see that $bz+ac=xy$. (Note that this equality follows from the Ptolemaeus theorem.) On the other hand, using the Pythagoras theorem for triangles $ABD$ and $ACD$, we obtain $ac+zb=\sqrt{(z^2-c^2)(z^2-a^2)}$. So $z^4-(a^2+b^2+c^2)z^2-2abcz=0$, and since $z\ne 0$, we have $$z^3-(a^2+b^2+c^2)z-2abc=0.$$ Note that the value of $z$ is independent of the lengths of the sides $AB$, $BC$ and $CD$, so without loss of generality, we set $a=2$, $b=7$, $c=11$. We have $$z^3-174z-308=0.(1)$$ We find one of the roots of this equation: $z=14$. Two other roots of (1) are negative numbers.

Using the sines law for the triangle $BCD$, we obtain $7=b=z\sin \angle BDC=14\sin \angle BDC$, so $\sin \angle BDC=1/2$, i.e. $\angle BDC=30^{\circ }$. Therefore the angle between the diagonals $AC$ and $BD$ is equal to $\angle COD=90^{\circ }-\angle BDC=90^{\circ }-30^{\circ }=60^{\circ }$. Now we find $$AC=x=\sqrt{z^2-c^2}=\sqrt{196-121}=\sqrt{75}=5\sqrt{3},$$ $$BD=y=\sqrt{z^2-a^2}=\sqrt{196-4}=\sqrt{192}=8\sqrt{3}.$$ Therefore the required area is equal to $$S(ABCD)=0.5\cdot AC\cdot BD\cdot \sin \angle COD=0.5\cdot 5\sqrt{3}\cdot 8\sqrt{3}\cdot \sqrt{3}/2=30\sqrt{3}.$$