We here examine the expressive power of first order logic with generalized quantifiers over finite ordered structures. In particular, we address the following problem: Given a family $\mathbf{Q}$ of generalized quantifiers expressing a complexity class $\mathbf{C}$, what is the expressive power of first order logic FO($\mathbf{Q}$) extended by the quantifiers in $\mathbf{Q}$? From previously studied examples, one would expect that FO($\mathbf{Q}$) captures $\mathbf{L}^\mathbf{C}$, i.e., logarithmic space relativized to an oracle in $\mathbf{C}$. We show that this is not always true. However, after studying the problem from a general point of view, we derive sufficient conditions on $\mathbf{C}$ such that FO($\mathbf{Q}$) captures $\mathbf{L}^\mathbf{C}$. These conditions are fulfilled by a large number of relevant complexity classes, in particular, for example, by $\mathbf{NP}$. As an application of this result, it follows that first order logic extended by Henkin quantifiers captures $\mathbf{L}^{\mathbf{NP}}$. This answers a question raised by Blass and Gurevich [Ann. Pure Appl. Logic, vol. 32, 1986]. Furthermore we show that for many families $\mathbf{Q}$ of generalized quantifiers (including the family of Henkin quantifiers), each FO($\mathbf{Q}$)-formula can be replaced by an equivalent FO($\mathbf{Q}$)-formula with only two occurrences of generalized quantifiers. This generalizes and extends an earlier normal-form result by I. A. Stewart [Fundamenta Inform. vol. 18, 1993].