Probability, Random Processes, and Statistical Analysis Together with the fundamentals of probability, random processes, and statistical analy- sis, this insightful book also presents a broad range of advanced topics and applications not covered in other textbooks. Advanced topics include: • Bayesian inference and conjugate priors • Chernoff bound and large deviation approximation • Principal component analysis and singular value decomposition • Autoregressive moving average (ARMA) time series • Maximum likelihood estimation and the Expectation-Maximization (EM) algorithm • Brownian motion, geometric Brownian motion, and Ito process • Black–Scholes differential equation for option pricing • Hidden Markov model (HMM) and estimation algorithms • Bayesian networks and sum-product algorithm • Markov chain Monte Carlo methods • Wiener and Kalman filters • Queueing and loss networks The book will be useful to students and researchers in such areas as communications, signal processing, networks, machine learning, bioinformatics, and econometrics and mathematical finance. With a solutions manual, lecture slides, supplementary materials, and MATLAB programs all available online, it is ideal for classroom teaching as well as a valuable reference for professionals. Hisashi Kobayashi is the Sherman Fairchild University Professor Emeritus at Princeton University, where he was previously Dean of the School of Engineering and Applied Science.
He also spent 15 years at the IBM Research Center, Yorktown Heights, NY, and was the Founding Director of IBM Tokyo Research Laboratory. He is an IEEE Life Fellow, an IEICE Fellow, was elected to the Engineering Academy of Japan (1992), and received the 2005 Eduard Rhein Technology Award. Mark is a Professor in the Department of Electrical and Computer Engineer- ing at George Mason University. Prior to this, he was a research staff member at the NEC C&C Research Laboratories in Princeton, New Jersey, and in 2002 he received a National Science Foundation CAREER award.
William Turin is currently a Consultant at AT&T Labs Research. As a Member of Tech- nical Staff at AT&T Bell Laboratories and later a Technology Consultant at AT&T Labs Research for 21 years, he developed methods for quantifying the performance of communication systems. He is the author of six books and numerous papers.com “This book provides a very comprehensive, well-written and modern approach to the fundamentals of probability and random processes, together with their applications in the statistical analysis of data and signals. It provides a one-stop, unified treatment that gives the reader an understanding of the models, methodologies, and underlying princi- ples behind many of the most important statistical problems arising in engineering and the sciences today.
Vincent Poor, Princeton University “This is a well-written, up-to-date graduate text on probabilty and random processes. It is unique in combining statistical analysis with the probabilistic material. As noted by the authors, the material, as presented, can be used in a variety of current application areas, ranging from communications to bioinformatics. I particularly liked the historical introduction, which should make the field exciting to the student, as well as the intro- ductory chapter on probability, which clearly describes for the student the distinction between the relative frequency and axiomatic approaches to probability.
I recommend it unhesitatingly. It deserves to become a leading text in the field.” Professor Emeritus Mischa Schwartz, Columbia University “Hisashi Kobayashi, Brian L. Mark, and William Turin are highly experienced uni- versity teachers and scientists. Based on this background, their book covers not only fundamentals, but also a large range of applications.
Some of them are treated in a textbook for the first time. Without any doubt the book will be extremely valuable to graduate students and to scientists in universities and industry. Congratulations to the authors!” Professor Dr. Eberhard Hänsler, Technische Universität Darmstadt “An up-to-date and comprehensive book with all the fundamentals in probability, ran- dom processes, stochastic analysis, and their interplays and applications, which lays a solid foundation for the students in related areas.
It is also an ideal textbook with five relatively independent but logically interconnected parts and the corresponding solution manuals and lecture slides. Furthermore, to my best knowledge, similar editing in Part IV and Part V can’t be found elsewhere.” Zhisheng Niu, Tsinghua University www.com Probability, Random Processes, and Statistical Analysis HISASHI KO BAYASHI Princeton University BRIAN L. MARK George Mason University WILLIAM TURIN AT&T Labs Research www.com CAMBRIDGE UNIVERSITY PRESS Cambridge, New York, Melbourne, Madrid, Cape Town Singapore, São Paulo, Delhi, Tokyo, Mexico City Cambridge University Press The Edinburgh Building, Cambridge CB2 8RU, UK Published in the United States of America by Cambridge University Press, New York www.org Information on this title: www.org/9780521895446 c Cambridge University Press 2012 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press.
First published 2012 Printed in the United Kingdom at the University Press, Cambridge A catalogue record for this publication is available from the British Library ISBN 978-0-521-89544-6 Hardback Additional resources for this publication at www.org/9780521895446 Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.com To Masae, Karen, and Galina www.com Contents List of abbreviations and acronyms page xviii Preface xxiii Acknowledgments xxx 1 Introduction 1 1.1 Why study probability, random processes, and statistical analysis? 1 1.1 Communications, information, and control systems 2 1.4 Biostatistics, bioinformatics, and related fields 4 1.5 Econometrics and mathematical finance 4 1.6 Queueing and loss systems 6 1.7 Other application domains 6 1.2 History and overview 7 1.1 Classical probability theory 7 1.2 Modern probability theory 9 1.4 Statistical analysis and inference 12 1.3 Discussion and further reading 14 Part I Probability, random variables, and statistics 15 2 Probability 17 2.1 Randomness in the real world 17 2.1 Repeated experiments and statistical regularity 17 2.2 Random experiments and relative frequencies 18 2.2 Axioms of probability 18 2.4 Properties of probability measure 21 www.com viii Contents 2.3 Bernoulli trials and Bernoulli’s theorem 26 2.4 Conditional probability, Bayes’ theorem, and statistical independence 30 2.1 Joint probability and conditional probability 30 2.3 Statistical independence of events 34 2.5 Summary of Chapter 2 36 2.6 Discussion and further reading 37 2.7 Problems 37 3 Discrete random variables 42 3.2 Two random variables and joint distribution function 44 3.2 Discrete random variables and probability distributions 45 3.1 Joint and conditional probability distributions 46 3.2 Moments, central moments, and variance 50 3.3 Covariance and correlation coefficient 51 3.3 Important probability distributions 53 3.1 Bernoulli distribution and binomial distribution 54 3.4 Negative binomial (or Pascal) distribution 59 3.5 Zipf’s law and zeta distribution 62 3.4 Summary of Chapter 3 65 3.5 Discussion and further reading 66 3.6 Problems 66 4 Continuous random variables 72 4.1 Continuous random variables 72 4.1 Distribution function and probability density function 72 4.2 Expectation, moments, central moments, and variance 73 4.2 Important continuous random variables and their distributions 75 4.3 Joint and conditional probability density functions 90 4.1 Bivariate normal (or Gaussian) distribution 92 4.2 Multivariate normal (or Gaussian) distribution 94 4.4 Exponential family of distributions 95 4.5 Bayesian inference and conjugate priors 97 www.com Contents ix 4.6 Summary of Chapter 4 103 4.7 Discussion and further reading 104 4.8 Problems 104 5 Functions of random variables and their distributions 112 5.1 Function of one random variable 112 5.2 Function of two random variables 115 5.3 Two functions of two random variables and the Jacobian matrix 119 5.4 Generation of random variates for Monte Carlo simulation 123 5.1 Random number generator (RNG) 124 5.2 Generation of variates from general distributions 125 5.3 Generation of normal (or Gaussian) variates 130 5.5 Summary of Chapter 5 131 5.6 Discussion and further reading 131 5.7 Problems 132 6 Fundamentals of statistical data analysis 138 6.1 Sample mean and sample variance 138 6.2 Relative frequency and histograms 140 6.1 Histogram on probability paper 142 6.2 Log-survivor function curve 144 6.3 Hazard function and mean residual life curves 148 6.4 Dot diagram and correlation coefficient 149 6.4 Summary of Chapter 6 152 6.5 Discussion and further reading 152 6.6 Problems 153 7 Distributions derived from the normal distribution 157 7.1 Chi-squared distribution 157 7.4 Log-normal distribution 165 7.5 Rayleigh and Rice distributions 167 7.6 Complex-valued normal variables 172 7.1 Complex-valued Gaussian variables and their properties 172 7.2 Multivariate Gaussian variables 173 7.7 Summary of Chapter 7 176 7.8 Discussion and further reading 177 7.com x Contents Part II Transform methods, bounds, and limits 183 8 Moment-generating function and characteristic function 185 8.1 Moment-generating function (MGF) 185 8.1 Moment-generating function of one random variable 185 8.2 Moment-generating function of sum of independent random variables 189 8.3 Joint moment-generating function of multivariate random variables 190 8.1 Characteristic function of one random variable 192 8.2 Sum of independent random variables and convolution 196 8.3 Moment generation from characteristic function 198 8.4 Joint characteristic function of multivariate random variables 199 8.5 Application of the characteristic function: the central limit theorem (CLT) 201 8.6 Characteristic function of multivariate complex-valued normal variables 202 8.3 Summary of Chapter 8 204 8.4 Discussion and further reading 205 8.5 Problems 206 9 Generating functions and Laplace transform 211 9.1 Probability-generating function (PGF) 212 9.2 Sum of independent variables and convolutions 215 9.3 Sum of a random number of random variables 217 9.4 Inverse transform of generating functions 218 9.2 Laplace transform method 226 9.1 Laplace transform and moment generation 226 9.2 Inverse Laplace transform 229 9.3 Summary of Chapter 9 234 9.4 Discussion and further reading 235 9.5 Problems 235 10 Inequalities, bounds, and large deviation approximation 241 10.1 Inequalities frequently used in probability theory 241 10.1 Cauchy–Schwarz inequality 241 10.3 Shannon’s lemma and log-sum inequality 246 10.4 Markov’s inequality 248 www.com Contents xi 10.6 Kolmogorov’s inequalities for martingales and submartingales 250 10.1 Chernoff’s bound for a single random variable 253 10.2 Chernoff’s bound for a sum of i.3 Large deviation theory 257 10.1 Large deviation approximation 257 10.2 Large deviation rate function 263 10.4 Summary of Chapter 10 267 10.5 Discussion and further reading 268 10.6 Problems 269 11 Convergence of a sequence of random variables and the limit theorems 277 11.1 Preliminaries: convergence of a sequence of numbers or functions 277 11.1 Sequence of numbers 277 11.2 Sequence of functions 278 11.2 Types of convergence for sequences of random variables 280 11.1 Convergence in distribution 280 11.2 Convergence in probability 282 11.3 Almost sure convergence 285 11.4 Convergence in the r th mean 288 11.5 Relations between the modes of convergence 292 11.1 Infinite sequence of events 294 11.2 Weak law of large numbers (WLLN) 298 11.3 Strong laws of large numbers (SLLN) 300 11.4 The central limit theorem (CLT) revisited 303 11.4 Summary of Chapter 11 306 11.5 Discussion and further reading 307 11.6 Problems 308 Part III Random processes 313 12 Random processes 315 12.2 Classification of random processes 316 12.1 Discrete-time versus continuous-time processes 316 12.2 Discrete-state versus continuous-state processes 317 12.3 Stationary versus nonstationary processes 317 12.4 Independent versus dependent processes 318 12.5 Markov chains and Markov processes 318 12.6 Point processes and renewal processes 321 www.com xii Contents 12.7 Real-valued versus complex-valued processes 321 12.8 One-dimensional versus vector processes 322 12.3 Stationary random process 322 12.1 Strict stationarity versus wide-sense stationarity 323 12.3 Ergodic processes and ergodic theorems 327 12.4 Complex-valued Gaussian process 329 12.1 Complex-valued Gaussian random variables 329 12.2 Complex-valued Gaussian process 330 12.3 Hilbert transform and analytic signal 333 12.5 Summary of Chapter 12 339 12.6 Discussion and further reading 340 12.7 Problems 340 13 Spectral representation of random processes and time series 343 13.1 Spectral representation of random processes and time series 343 13.3 Analysis of periodic wide-sense stationary random process 346 13.5 Power spectrum and periodogram of time series 350 13.2 Generalized Fourier series expansions 357 13.1 Review of matrix analysis 357 13.2 Karhunen–Loève expansion and its applications 363 13.3 Principal component analysis and singular value decomposition 372 13.