The asymptotic distribution of an $M$-estimator is studied when the underlying distribution is discrete. Asymptotic normality is shown to hold quite generally within the assumed parametric family. When the specification of the model is inexact, however, it is demonstrated that an $M$-estimator whose score function is not everywhere differentiable, e.g., a Huber estimator, has a nonnormal limiting distribution at certain distributions, resulting in unstable inference in the neighborhood of such distributions. Consequently, smooth score functions are proposed for discrete data.