We study closures of ${\rm GL}^+_2(\mathbb{R})$ -orbits in the total space $\Omega M_g$ of the Hodge bundle over the moduli space of curves under the assumption that they are algebraic manifolds. We show that in the generic stratum, such manifolds are the whole stratum, the hyperelliptic locus, or parameterize curves whose Jacobian has additional endomorphisms. This follows from a cohomological description of the tangent bundle to $\Omega M_g$ . For nongeneric strata, similar results can be shown by a case-by-case inspection. We also propose to study a notion of linear manifold that comprises Teichmüller curves, Hilbert modular surfaces, and the ball quotients of Deligne and Mostow [DM]. Moreover, we give an explanation for the difference between Hilbert modular surfaces and Hilbert modular threefolds with respect to this notion of linearity