THE 68th ROMANIAN MATHEMATICAL OLYMPIAD

Let $f$, $g$, $h$ be three functions satisfying the properties in the problem. Because $g\circ h=1_B$, it is clear that $g$ is surjective and $h$ is injective. As $f=h\circ g$, we also get that $|B|=|\text{Im }f|\le |A|$.

As $f(f(x))=h(g(h(g)))=h(g(x))=f(x)$ we get $y\in \text{Im }f$, $f(y)=y$. From here, any function $f$ with the property from the problem has the following form $$f(x)=\{\begin{array}{ll}x,& \text{if }x\in \text{Im }f\\ \phi (x),& \text{if }x\in A\setminus \text{Im }f\end{array}$$ where $\phi :A\setminus \text{Im }f\to \text{Im }f$ is any function.

Let $A^{\prime }\subseteq A$, such that $|A^{\prime }|=|B|$ (supposing that $|B|\le |A|$). We define $$f(x)=\{\begin{array}{ll}x,& \text{if }x\in A^{\prime }\\ \phi (x),& \text{if }x\in A\setminus A^{\prime }\end{array}$$ where $\phi :A\setminus A^{\prime }\to A^{\prime }$ is any function.

Noticing that $\text{Im }f=A^{\prime }$ we prove that $f$ has the property from the problem. Let $h:B\to A^{\prime }\subseteq A$ an arbitrary bijection and $g:A\to B$ defined by $$g(x)=\{\begin{array}{ll}{h}^{-1}(x),& \text{if }x\in A^{\prime }\\ h^{-1}(\phi (x)),& \text{if }x\in A\setminus A^{\prime }.\end{array}$$ It is easy to check that $g(h(x))=x$, $\forall x\in B$, and $h(g(x))=f(x)$, $\forall x\in A$.

Now we can choose the subset $A^{\prime }$ in $\left(\genfrac{}{}{0ex}{}{|A|}{|B|}\right)$ ways, and the function $\phi$ in $|B{|}^{|A|-|B|}$ ways, and thus, the number of functions with the requested property is $\left(\genfrac{}{}{0ex}{}{|A|}{|B|}\right)|B{|}^{|A|-|B|}$.