We prove that the extended Toda hierarchy of \cite{CDZ} admits nonabelian Lie
algebra of infinitesimal symmetries isomorphic to the half of the Virasoro
algebra. The generators $L_m$, $m\geq -1$ of the Lie algebra act by linear
differential operators onto the tau function of the hierarchy. We also prove
that the tau function of a generic solution to the extended Toda hierarchy is
annihilated by a combination of the Virasoro operators and the flows of the
hierarchy. As an application we show that the validity of the Virasoro
constraints for the $CP^1$ Gromov-Witten invariants and their descendents
implies that their generating function is the logarithm of a particular tau
function of the extended Toda hierarchy.