jmc

We can substitute $5$, $3$, and $6$ into the functions' definitions: $$\begin{array}{rl}(5⧫3)★6& =\left(5^2-3^2\right)★6\\ (5⧫3)★6& =\left(25-9\right)★6\\ & =16★6\\ & =(16-6{)}^2\\ & =\boxed{100}.\end{array}$$