VMO

We will prove that the maximal value of the given expression $P$ is $\frac{3}{16}$. Indeed, applying these inequalities $$x^4+y^4\ge xy(x^2+y^2)\text{ and }(xy+z^2{)}^2\ge 4xy{z}^2,$$ we have $$\begin{gathered} (x^4+y^4)(xy+z^2{)}^3\ge 4x^2{y}^2{z}^2(x^2+y^2)(xy+z^2) \\ \ge 4x^2{y}^2{z}^2(z^2{x}^2+z^2{y}^2+2x^2{y}^2). \end{gathered}$$ Thus, we have $$\begin{gathered} \frac{x^3{y}^4{z}^3}{(x^4+y^4)(xy+z^2{)}^3}\le \frac{x^3{y}^4{z}^3}{4x^2{y}^2{z}^2(z^2{x}^2+z^2{y}^2+2x^2{y}^2)} \\ =\frac{x{y}^{2}z}{4(z^2{x}^2+z^2{y}^2+2x^2{y}^2)} \end{gathered}$$ We will prove that $\sum \frac{x{y}^{2}z}{z^2{x}^2+z^2{y}^2+2x^2{y}^2}\le \frac{3}{4}$. Put $a=xy$, $b=yz$ and $c=zx$, the left hand side becomes $$\sum \frac{ab}{2a^2+b^2+c^2}\le \frac{3}{4}.$$ If $a\ge b\ge c$ then $ab\ge ac\ge bc$ and $$\frac{1}{2c^2+a^2+b^2}\ge \frac{1}{2b^2+c^2+a^2}\ge \frac{1}{2a^2+b^2+c^2}.$$ Applying the rearrangement inequality, we have $$\sum \frac{ab}{2a^2+b^2+c^2}\le \sum \frac{ab}{2c^2+a^2+b^2}.$$ By AM-GM and Cauchy-Schwarz inequalities, we obtain that $$\begin{array}{rl}4\sum \frac{ab}{2c^2+a^2+b^2}& \le \sum \frac{(a+b{)}^2}{2c^2+a^2+b^2}\\ & \le \sum \left(\frac{a^2}{c^2+a^2}+\frac{b^2}{c^2+b^2}\right)=3.\end{array}$$ $$\text{Hence, }P=\sum \frac{x^3{y}^4{z}^3}{(x^4+y^4)(xy+z^2{)}^3}\le \frac{3}{16}.$$ The equality holds if and only if $a=b=c$ or $x=y=z$. Therefore, the maximal value of the given expression is $\frac{3}{16}$. $\square$