jmc

Let $p=ab$. We add the two given equations, giving $$\begin{array}{rl}({\log }_{8}a+{\log }_4{b}^2)+({\log }_{8}b+{\log }_4{a}^2)& =12\\ {\log }_8(ab)+{\log }_4(a^2{b}^2)& =12\\ {\log }_{8}p+{\log }_4{p}^2& =12\\ {\log }_{8}p+2{\log }_{4}p& =12.\end{array}$$Using the change-of-base formula, we have $${\log }_{8}p=\frac{{\log }_{4}p}{{\log }_{4}8}=\frac{{\log }_{4}p}{3/2}=\frac{2}{3}{\log }_{4}p,$$so we can write both logarithms in base $4$: $$\frac{2}{3}{\log }_{4}p+2{\log }_{4}p=12,$$or $\frac{8}{3}{\log }_{4}p=12$. Thus, ${\log }_{4}p=12\cdot \frac{3}{8}=\frac{9}{2}$, so $$p=4^{9/2}=2^9=\boxed{512}.$$