We provide an analog of the Joyal-Street center construction and of the Kassel-Turaev categorical quantum double in the context of the crossed categories introduced by Turaev. Then, we focus or attention to the case of categories of representation. In particular, we introduce the notion of a Yetter-Drinfeld module over a crossed group coalgebra H and we prove that both the category of Yetter-Drinfeld modules over H and the center of the category of representations of H as well as the category of representations of the quantum double of H are isomorphic as braided crossed categories.