China Western Mathematical Olympiad

When $x=y=\frac{1}{3}$, the value of the expression is $\frac{1}{8}$.

We will prove that $\frac{xy(1-x-y)}{(x+y)(1-x)(1-y)}\le \frac{1}{8}$ for any $0<x,y<1$ as follows.

If $x+y\ge 1$, then $\frac{xy(1-x-y)}{(x+y)(1-x)(1-y)}\le 0<\frac{1}{8}$.

If $x+y<1$, then let $1-x-y=z>0$, it follows from the AM-GM Inequality that $$\begin{gathered} \frac{xy(1-x-y)}{(x+y)(1-x)(1-y)}=\frac{xyz}{(x+y)(y+z)(z+x)} \\ \le \frac{xyz}{2\sqrt{xy}\cdot 2\sqrt{yz}\cdot 2\sqrt{zx}}=\frac{1}{8}. \end{gathered}$$ In conclusion, the maximum value of the expression is $\frac{1}{8}$.