We give a necessary and sufficient condition, of geometric type, for the uniform decay of energy of solutions of the linear system of magnetoelasticity in a bounded domain with smooth boundary. A Dirichlet-type boundary condition is assumed. When the geometrical condition is not fulfilled, we show polynomial decay of the energy, for smooth initial conditions. Our strategy is to use micro-local defect measures to show suitable observability inequalities on high-frequency solutions of the Lamé system.