For a fixed odd prime $p$ and a representation $\varrho$ of the absolute Galois
group of $\mathbb{Q}$ into the projective group ${\rm PGL}_2(\mathbb{F}_p)$, we
provide the twisted modular curves whose rational points supply the quadratic
$\mathbb{Q}$-curves of degree $N$ prime to $p$ that realize $\varrho$ through
the Galois action on their $p$-torsion modules. The modular curve to twist is
either the fiber product of $X_0(N)$ and $X(p)$ or a certain quotient of
Atkin-Lehner type, depending on the value of $N$ mod $p$. For our purposes, a
special care must be taken in fixing rational models for these modular curves
and in studying their automorphisms. By performing some genus computations, we
obtain as a by-product some finiteness results on the number of quadratic
$\mathbb{Q}$-curves of a given degree $N$ realizing $\varrho$.