Irska

Proceed as in the hint and let $P^{\prime }$ denote the rotated position of $P$. Thus $|A{P}^{\prime }|=3$, $|B{P}^{\prime }|=5$ and $\angle PA{P}^{\prime }=90^{\circ }$. Since $PA{P}^{\prime }$ is an isosceles right triangle with $|A{P}^{\prime }|=|AP|=3$, then $|P{P}^{\prime }|=3\sqrt{2}$.



The area of the triangle $PB{P}^{\prime }$ is, by Heron's formula, given by $$\sqrt{\left(\frac{3\sqrt{2}+12}{2}\right)\left(\frac{3\sqrt{2}-2}{2}\right)\left(\frac{3\sqrt{2}+2}{2}\right)\left(\frac{12-3\sqrt{2}}{2}\right)}=\frac{21}{2}$$ But the area of the triangle $PB{P}^{\prime }$ is also given by $\frac{1}{2}|PB||P{P}^{\prime }|\sin \angle BP{P}^{\prime }$, which is equal to $\frac{21\sqrt{2}}{2}\sin \angle BP{P}^{\prime }$ and so $\frac{1}{\sqrt{2}}=\sin \angle BP{P}^{\prime }$. This gives $\angle BP{P}^{\prime }=45^{\circ }$, and so $\angle APB=90^{\circ }$. From the right triangle $APB$ we get $|AB{|}^2=58$. Thus the area of the square is $58$.