This paper discusses the behavior of the maximum likelihood estimator (MLE), in the case that the true parameter cannot be identified uniquely. Among many statistical models with unidentifiability, neural network models are the main concern of this paper. It is known in some models with unidentifiability that the asymptotics of the likelihood ratio of the MLE has an unusually larger order. Using the framework of locally conic models put forth by Dacunha-Castelle and Gassiat as a generalization of Hartigan's idea, a useful sufficient condition of such larger orders is derived. This result is applied to neural network models, and a larger order is proved if the true function is given by a smaller model. Also, under the condition that the model has at least two redundant hidden units, a log n lower bound for the likelihood ratio is derived.