We study a new class of infinite-dimensional Lie algebras W_\infty(p,q)
generalizing the standard W_\infty algebra, viewed as a tensor operator algebra
of SU(1,1) in a group-theoretic framework. Here we interpret W_\infty(p,q)
either as an infinite continuation of the pseudo-unitary symmetry U(p,q), or as
a "higher-U(p,q)-spin extension" of the diffeomorphism algebra diff(p,q) of the
N=p+q torus U(1)^N. We highlight this higher-spin structure of W_\infty(p,q) by
developing the representation theory of U(p,q) (discrete series), calculating
higher-spin representations, coherent states and deriving K\"ahler structures
on flag manifolds. They are essential ingredients to define operator symbols
and to infer a geometric pathway between these generalized W_\infty symmetries
and algebras of symbols of U(p,q)-tensor operators. Classical limits (Poisson
brackets on flag manifolds) and quantum (Moyal) deformations are also
discussed. As potential applications, we comment on the formulation of
diffeomorphism-invariant gauge field theories, like gauge theories of
higher-extended objects, and non-linear sigma models on flag manifolds.