Let $(\xi,\eta)$ be a bivariate Lévy process such that the integral $\int_0^\infty e^{-\xi_{t-}} \, d\eta_t$ converges almost surely. We characterise, in terms of their \LL measures, those Lévy processes for which (the distribution of) this integral has atoms. We then turn attention to almost surely convergent integrals of the form $I:=\int_0^\infty g(\xi_t) \, dt$, where $g$ is a deterministic function. We give sufficient conditions ensuring that $I$ has no atoms, and under further conditions derive that $I$ has a Lebesgue density. The results are also extended to certain integrals of the form $\int_0^\infty g(\xi_t) \, dY_t$, where $Y$ is an almost surely strictly increasing stochastic process, independent of $\xi$.