We prove the existence of a local smooth Levi decomposition for smooth Poisson structures and Lie algebroids near a singular point. This Levi decomposition is a kind of normal form or partial linearization, which was established in the formal case by Wade [10] and in the analytic case by the second author [15]. In particular, in the case of smooth Poisson structures with a compact semisimple linear part, we recover Conn's smooth linearization theorem [5], and in the case of smooth Lie algebroids with a compact semisimple isotropy Lie algebra, our Levi decomposition result gives a positive answer to a conjecture of Weinstein [13] on the smooth linearization of such Lie algebroids. In the appendix of this paper, we show an abstract Nash-Moser normal form theorem, which generalizes our Levi decomposition result, and which may be helpful in the study of other smooth normal form problems.