In practice, we are often interested in predicting the value of a physical quantity; for example, we would like to know how the radiation of a radioactive sample will change with time, or how the electric current produced by a thermocouple will change if we change the temperature. In many such situations, we have a formula, a model that describes how the quantity of interest (amount of radiation or current) depends on the easier-to-predict quantities, such as time or temperature. However, the corresponding formulas usually contain unknown parameters whose values must be determined experimentally. For a simple radioactive sample, the level of radiation’s dependence on time is exponential, but the parameters of this exponential function must be experimentally determined. In other cases, we know that the dependence is linear, but the exact parameters of this linear dependence must be determined from the experimental data.
In the ideal situation, when the measurement errors can be safely ignored, every measurement result leads to an equation that contains the unknown parameters. Once we perform as many measurements as there are parameters, we can solve the corresponding system of equations and determine the values of the parameters, usually uniquely. In many situations, the resulting algorithms are computationally simple; they can be easily implemented on a simple calculator or even on the back of an envelope.
Usually, the measurement errors cannot be ignored. However, practitioners often ignore these errors and use simple no-error algorithms to estimate the values of the corresponding parameters. This approach made sense in the early twentieth century, when computation was difficult and simple estimates were all one could do in the field. Nowadays, computers are ubiquitous, so it makes perfect sense to use more complex algorithms, if these algorithms lead to much more accurate estimates. In the first chapters of the book, the author lists convincing examples that the accuracy of computationally simple no-error algorithms is often much worse than the accuracy of the more computationally complex statistical algorithms.
When practitioners take into account measurement uncertainty, they mostly use techniques they learned in Statistics 101--the least-squares method. The least-squares method is optimal for parameter estimation under the assumption that all the measurement errors are independent and normally distributed with the same standard deviation. It is under this assumption that this method was first derived by Gauss (the least-squares method is also optimal in a few somewhat more general situations). In practice, however, the above assumption is often not satisfied: different measurement errors are often correlated, have different standard deviations, and their distribution is often not Gaussian. The author convincingly shows that, in such situations, the least-square estimates can be drastically improved.
The main objective of this book is to encourage practitioners to use better parameter-estimation techniques. The book starts with explaining the advantages of statistical parameter-estimation methods, in chapter 2. Chapter 3 describes different distributions, both Gaussian and non-Gaussian, that are frequently encountered in practice. Before the author starts explaining how to estimate parameters, he describes, in chapter 4, lower bounds on the precision and accuracy of the estimates--mostly, different versions of Cramer-Rao bounds. Parameter-estimation methods--based on maximum likelihood--are described in chapter 5. Chapter 5 also contains the basics of hypothesis testing: whether our original assumptions about the probability distributions are in good accordance with the experimental results. Finally, chapter 6 describes different numerical optimization techniques and how they can be used to find the maximum likelihood estimates. In particular, the author spends some time explaining the Levenberg-Marquardt method in which a special “regularization” idea is used to avoid singular matrices and related numerical problems.
The book is very well and convincingly written. The author indicates, rather cautiously, that some basic familiarity with statistics is expected; in reality, many basic statistical results and notations, as well as results about vectors and matrices, are repeated in the appendices. Most results are given with proofs that are usually rather simple and easy to follow. On the other hand, proofs are clearly marked, so readers who only want to learn the techniques and algorithms--or who are scared of the proofs--can easily skip them. Every chapter has exercises; solutions to some of these exercises form chapter 7.
The author’s objective was to write a useful and, thus, not very long book. This book is definitely not an encyclopedia; with 288 pages, it has to be rather narrowly focused. Readers interested in related topics such as robust estimates--what to do when we do not know the exact shape of the corresponding probability distributions--should look into more comprehensive handbooks. This conciseness is sometimes achieved at the expense of detailed explanations of the algorithm details: for example, in the description of the gradient descent methods, the author never explains how exactly to reduce the step, when the original step size does not decrease the value of the minimized objective function.
Minor recommendations to the readers are in order. First, matrix notations should not intimidate readers; these notations are well explained in Appendix B. (I wish the author put more emphasis on the existence of this appendix in the beginning of the book.) The electrical engineering notation j is used for the square root of 1; this may be somewhat confusing to physicists who are more accustomed to the i notation. It may also be somewhat confusing to practitioners: the meanings of some terms in statistics are different from the meanings of these terms in other computations-related disciplines. For example, in statistical estimates, “precision” simply means bias, while “accuracy” means standard deviation, and both terms have nothing to do with the number of digits--as in “single” or “double” precision. Also, in statistical estimates, “consistency” does not mean the absence of a contradiction, as in logic, but that the estimate tends to the actual value when the sample size increases. It may also be somewhat confusing that here we have two different notions of independence for random variables: the more familiar notion of statistical independence, and the notion of linear independence, with which most readers are familiar from linear (or matrix) algebra. A Greek letter kappa on page 182 should not be confused with the letter k.
It might have been a good idea to emphasize the relation between the new material of the book and what readers may have learned in their statistics classes. For example, the notion of the Fisher score--definitely unfamiliar to most readers--is introduced on page 22, and only on page 100 is it shown to be closely related with the notion of maximum likelihood--with which many readers are somewhat familiar. Similarly, on page 35, the author describes the Lorentz line without mentioning its relation to the Cauchy distribution--the distribution that many statistics textbooks cite as an example of a distribution with infinite standard deviation.
Overall, the book is very well written. I noticed only two points where nonmathematician readers--the intended audience--may be unclear: the mention of integrable functions on page 51 and chi-square on page 127.
I highly recommend this book to practitioners who want to systematically learn and use new, better techniques for parameter estimation. With numerous exercises, this book can also be recommended as a very good textbook.
