For stationary solutions of the affine stochastic delay differential equation \d X(t) =\left(\gamma_0X(t)+\gamma_rX(t-r)+\int_{-r}^0 X(t+u)g(u)\d u\right)\d t+\sigma\d W(t), we consider the problem of nonparametric inference for the weight function g and for γ0,γr from the continuous observation of one trajectory up to time T>0. For weight functions in the scale of Besov spaces Bsp,1 and Lρ-type loss functions, convergence rates are established for long-time asymptotics. The estimation problem is equivalent to an ill-posed inverse problem with error in the data and unknown operator. We propose a wavelet thresholding estimator that achieves the rate (T/logT)-s/(2s+3) under certain restrictions on p and ρ. This rate is shown to be optimal in a minimax sense.