Three types of asymptotic $\chi^2$ goodness-of-fit tests derived under the normal assumption have been used widely in factor analysis. Asymptotic behavior of the test statistics is investigated here for the factor analysis model with linearly or nonlinearly restricted factor loadings under weak assumptions on the factor vector and the error vector. In particular the limiting $\chi^2$ result for the three tests is shown to hold for the factor vector, either fixed or random with any distribution having finite second-order moments, and for the error vector with any distribution having finite second-order moments, provided that the components of the error vector are independent, not just uncorrelated. As special cases the result holds for exploratory and confirmatory factor analysis models and for certain nonnormal structural equation (LISREL) models.