Optimal estimators in learning theory

V. N. Temlyakov

Banach Center Publications, Tome 72 (2006), p. 341-366 / Harvested from The Polish Digital Mathematics Library

Résumé

This paper is a survey of recent results on some problems of supervised learning in the setting formulated by Cucker and Smale. Supervised learning, or learning-from-examples, refers to a process that builds on the base of available data of inputs $x_i$ and outputs $y_i$, i = 1,...,m, a function that best represents the relation between the inputs x ∈ X and the corresponding outputs y ∈ Y. The goal is to find an estimator $f_{z}$ on the base of given data $z: = ((x₁,y₁),...,(x_m,y_m))$ that approximates well the regression function $f_ρ$ of an unknown Borel probability measure ρ defined on Z = X × Y. We assume that $(x_i,y_i)$, i = 1,...,m, are indepent and distributed according to ρ. We discuss a problem of finding optimal (in the sense of order) estimators for different classes Θ (we assume $f_ρ ∈ Θ$). It is known from the previous works that the behavior of the entropy numbers ϵₙ(Θ,B) of Θ in a Banach space B plays an important role in the above problem. The standard way of measuring the error between a target function $f_ρ$ and an estimator $f_{z}$ is to use the $L₂(ρ_X)$ norm ($ρ_X$ is the marginal probability measure on X generated by ρ). The usual way in regression theory to evaluate the performance of the estimator $f_{z}$ is by studying its convergence in expectation, i.e. the rate of decay of the quantity $E(||f_{ρ} - f_{z}||²_{L₂(ρ_X)})$ as the sample size m increases. Here the expectation is taken with respect to the product measure $ρ^m$ defined on $Z^m$. A more accurate and more delicate way of evaluating the performance of $f_{z}$ has been pushed forward in [CS]. In [CS] the authors study the probability distribution function $ρ^m{z: ||f_{ρ} - f_{z}||_{L₂(ρ_X)} ≥ η}$ instead of the expectation $E(||f_{ρ} - f_{z}||²_{L₂(ρ_X)})$. In this survey we mainly discuss the optimization problem formulated in terms of the probability distribution function.

Détail

Publié le : 2006-01-01

Zbl 1102.62039

EUDML-ID : urn:eudml:doc:282344