In many studies, the scientific objective can be formulated in terms of a statistical model indexed by parameters, only some of which are of scientific interest. The other "nuisance parameters" are required to complete the specification of the probability mechanism but are not of intrinsic value in themselves. It is well known that nuisance parameters can have a profound impact on inference. Many approaches have been proposed to eliminate or reduce their impact. In this paper, we consider two situations: where the likelihood is completely specified; and where only a part of the random mechanism can be reasonably assumed. In either case, we examine methods for dealing with nuisance parameters from the vantage point of parameter estimating functions. To establish a context, we begin with a review of the basic concepts and limitations of optimal estimating functions. We introduce a hierarchy of orthogonality conditions for estimating functions that helps to characterize the sensitivity of inferences to nuisance parameters. It applies to both the fully and partly parametric cases. Throughout the paper, we rely on examples to illustrate the main ideas.