Let $(\Omega, \mathscr{F})$ be the measurable space consisting of $\Omega$, the set of sequences $(x_1, x_2, \cdots)$ from a finite set $A$, and $\mathscr{F}$, the usual product sigma-field. Let $X_1, X_2, \cdots$ be the usual coordinate random variables defined on $\Omega$. For $n = 1,2, \cdots$, let $\mathscr{F}_n$ be the sub sigma-field of $\mathscr{F}$ generated by $X_1, X_2, \cdots, X_n$. We prove the following: if $P$ is a probability measure on $\mathscr{F}$ stationary with respect to the one-sided shift transformation on $\Omega$ and if $N$ is a positive integer, then there is a periodic measure $Q$ on $\mathscr{F}$ such that $Q = P$ over $\mathscr{F}_N$. This is a stronger result than the known fact that the periodic measures are dense in the set of stationary measures under the weak topology. We also show that if $P$ assigns positive measure to every non-empty set in $\mathscr{F}_N$, it is possible to find an ergodic measure $Q$ such that $P = Q$ over $\mathscr{F}_N$. We investigate the entropies of all such ergodic measures $Q$ which approximate $P$ in this sense, and show that there is a unique ergodic measure $Q$ of maximal entropy such that $P = Q$ over $\mathscr{F}_N$.