If $X_1, \cdots, X_n$ is a random sample from $F(x - \theta)$, where $F$ is an unknown member of a specified class $\mathscr{F}$ of approximately normal symmetric distributions, then an $M$-estimator of the unknown location parameter $\theta$ is obtained by solving the equation $\sum^n_{i=1} \psi(X_i - \hat{\theta}_n) = 0$ for $\hat{\theta}_n$. A suitable measure of the robustness of the $M$-estimator is $\sup \{V(\psi, F): F \in \mathscr{F}\}$, where $V(\psi, F) = \int \psi^2 dF/(\int \psi' dF)^2$ is (under regularity conditions) the asymptotic variance of $n^{\frac{1}{2}}(\hat{\theta}_n - \theta)$. A necessary and sufficient condition for $F_0$ in $\mathscr{F}$ to maximize $V(\psi, F)$ is obtained, and the result is specialized to evaluate $\sup \{V(\psi, F):F \in \mathscr{F}\}$ when the model for $\mathscr{F}$ is the gross errors model or the Kolmogorov model.