A new mathematical object called a preferred point geometry is introduced in order to (a) provide a natural geometric framework in which to do statistical inference and (b) reflect the distinction between homogeneous aspects (e.g., any point $\theta$ may be the true parameter) and preferred point ones (e.g., when $\theta_0$ is the true parameter). Although preferred point geometry is applicable generally in statistics, we focus here on its relationship to statistical manifolds, in particular to Amari's expected geometry. A symmetry condition characterises when a preferred point geometry both subsumes a statistical manifold and, simultaneously, generalises it to arbitrary order. There are corresponding links with Barndorff-Nielsen's strings. The rather unnatural mixing of metric and nonmetric connections in statistical manifolds is avoided since all connections used are shown to be metric. An interpretation of duality of statistical manifolds is given in terms of the relation between the score vector and the maximum likelihood estimate.