Semi-parametric models typically involve a finite-dimensional parameter of interest ${\pmb\theta}\in{\pmb\Theta}\subseteq\mathbb{R}^k$, along with an infinite-dimensional nuisance parameter~$f$. Quite often, the submodels corresponding to a fixed value of ${\pmb\theta}$ possess a group structure that induces a maximal invariant $\sigma$-field ${\cal B}({\pmb\theta})$. In classical examples, where $f$ denotes the density of some independ\-ent and identically distributed innovations, ${\cal B}({\pmb\theta})$ is the $\sigma$-field generated by the ranks of the residuals associated with the parameter value ${\pmb\theta}$. It is shown that semi-parametrically efficient distribution-free inference procedures can generally be constructed from parametrically optimal ones by conditioning on ${\cal B}({\pmb\theta})$; this implies, for instance, that semi-parametric efficiency (at given $\pmb\theta$ and $f$) can be attained by means of rank-based methods. The same procedures, when combined with a consistent estimation of the underlying nuis\-ance density $f$, yield conditionally distribution-free semi-parametrically efficient inference methods, for example, semi-parametrically efficient permutation tests. Remarkably, this is achieved without any explicit tangent space or efficient score computations, and without any sample-splitting device. By means of several examples, including both i.i.d. and time-series models, we show how these results apply in models for which rank-based inference or permutation tests have so far seldom been considered.