48th Austrian Mathematical Olympiad

a. The given equation and the AM-GM-inequality imply $$4\sqrt{xyz}=(x+y)(x+z)=x(x+y+z)+yz\ge 2\sqrt{xyz(x+y+z)}.$$ Therefore, $2\ge \sqrt{x+y+z}$ which gives $4\ge x+y+z$. Since we will explicitly give infinitely many triples with $x+y+z=4$ in the second part, $M=4$ is the maximum.

b. For $x+y+z=4$, equality must hold in the AM-GM-inequality of the first part, so we have $x(x+y+z)=yz$ and also $x+y+z=4$. If we choose $y=t$ with rational $t$ we get $4x=t(4-x-t)$ and therefore $x=\frac{4t-t^2}{4+t}$ and $z=4-x-y=\frac{16-4t}{4+t}$. If we take $0<t<4$ then all these expressions are positive and rational and are a solution of the given equation. The triples $(\frac{4t-t^2}{4+t},t,\frac{16-4t}{4+t})$ with rational $0<t<4$ are infinitely many cases of equality.