Recursive self-tuning control of finite Markov chains
Borkar, Vivek
Applicationes Mathematicae, Tome 24 (1997), p. 169-188 / Harvested from The Polish Digital Mathematics Library
Access to full text
Full (PDF)

A recursive self-tuning control scheme for finite Markov chains is proposed wherein the unknown parameter is estimated by a stochastic approximation scheme for maximizing the log-likelihood function and the control is obtained via a relative value iteration algorithm. The analysis uses the asymptotic o.d.e.s associated with these.

Publié le : 1997-01-01
EUDML-ID : urn:eudml:doc:219160 
@article{bwmeta1.element.bwnjournal-article-zmv24i2p169bwm,
     author = {Vivek Borkar},
     title = {Recursive self-tuning control of finite Markov chains},
     journal = {Applicationes Mathematicae},
     volume = {24},
     year = {1997},
     pages = {169-188},
     zbl = {0951.93537},
     language = {en},
     url = {http://dml.mathdoc.fr/item/bwmeta1.element.bwnjournal-article-zmv24i2p169bwm}
}

Borkar, Vivek. Recursive self-tuning control of finite Markov chains. Applicationes Mathematicae, Tome 24 (1997) pp. 169-188. http://gdmltest.u-ga.fr/item/bwmeta1.element.bwnjournal-article-zmv24i2p169bwm/

[000] [1] D. Bertsekas, Dynamic Programming--Deterministic and Stochastic Models, Prentice-Hall, Englewood Cliffs, N.J., 1987. | Zbl 0649.93001

[001] [2] V. S. Borkar, Identification and adaptive control of Markov chains, Ph.D. Thesis, Dept. of Electrical Engrg. and Computer Science, Univ. of California, Berkeley, 1980. | Zbl 0491.93063

[002] [3] V. S. Borkar, Topics in Controlled Markov Chains, Pitman Res. Notes in Math. 240, Longman Scientific and Technical, Harlow, 1991. | Zbl 0725.93082

[003] [4] V. S. Borkar, The Kumar-Becker-Lin scheme revisited, J. Optim. Theory Appl. 66 (1990), 289-309. | Zbl 0682.93060

[004] [5] V. S. Borkar, On Milito-Cruz adaptive control scheme for Markov chains, ibid. 77 (1993), 385-393. | Zbl 0791.93055

[005] [6] V. S. Borkar and K. Soumyanath, A new analog parallel scheme for fixed point computation I--theory, submitted. | Zbl 0953.65038

[006] [7] V. S. Borkar and P. P. Varaiya, Adaptive control of Markov chains I: finite parameter case, IEEE Trans. Automat. Control AC-24 (1979), 953-957. | Zbl 0416.93065

[007] [8] V. S. Borkar and P. P. Varaiya, Identification and adaptive control of Markov chains, SIAM J. Control Optim. 20 (1982), 470-488. | Zbl 0491.93063

[008] [9] Y.-S. Chow and H. Teicher, Probability Theory: Independence, Interchangeability, Martingales, Springer, New York, 1979.

[009] [10] B. Doshi and S. Shreve, Randomized self-tuning control of Markov chains, J. Appl. Probab. 17 (1980), 726-734. | Zbl 0442.93054

[010] [11] Y. El Fattah, Recursive algorithms for adaptive control of finite Markov chains, IEEE Trans. Systems Man Cybernet. SMC-11 (1981), 135-144.

[011] [12] --, Gradient approach for recursive estimation and control in finite Markov chains, Adv. Appl. Probab. 13 (1981), 778-803. | Zbl 0475.60051

[012] [13] M. Hirsch, Convergent activation dynamics in continuous time networks, Neural Networks 2 (1987), 331-349.

[013] [14] A. Jalali and M. Ferguson, Adaptive control of Markov chains with local updates, Systems Control Lett. 14 (1990), 209-218. | Zbl 0699.93047

[014] [15] P. R. Kumar and A. Becker, A new family of adaptive optimal controllers for Markov chains, IEEE Trans. Automat. Control AC-27 (1982), 137-142. | Zbl 0471.93069

[015] [16] P. R. Kumar and W. Lin, Optimal adaptive controllers for Markov chains, ibid., 756-774. | Zbl 0488.93036

[016] [17] H. Kushner and D. Clark, Stochastic Approximation for Constrained and Unconstrained Systems, Springer, Berlin, 1978. | Zbl 0381.60004

[017] [18] P. Mandl, Estimation and control in Markov chains, Adv. Appl. Probab. 6 (1974), 40-60. | Zbl 0281.60070

[018] [19] R. Milito and J. B. Cruz Jr., An optimization oriented approach to adaptive control of Markov chains, IEEE Trans. Automat. Control AC-32 (1987), 754-762. | Zbl 0632.93080

[019] [20] J. Neveu, Discrete-Parameter Martingales, North-Holland, Amsterdam, 1975.

[020] [21] B. Sagalovsky, Adaptive control and parameter estimation in Markov chains: a linear case, IEEE Trans. Automat. Control AC-27 (1982), 414-417. | Zbl 0479.93061

[021] [22] Ł. Stettner, On nearly self-optimizing strategies for a discrete-time uniformly ergodic adaptive model, Appl. Math. Optim. 27 (1993), 161-177. | Zbl 0769.93084

[022] [23] T. Yoshizawa, Stability Theory by Liapunov's Second Method, The Mathematical Society of Japan, 1966.