The Derivation

We want to derive an optimal estimate, $\hat{x}$, of a field, x, as a linear combination of observation data, $\theta$,

$$\hat{x} = A \theta$$ (1)

Our goal is to derive the form of the matrix, A, so that the expected mean square difference between the estimated field and the actual field ( x ) is minimized,

$$E[\varepsilon \varepsilon^T]= E[(\hat{x} - x)(\hat{x} - x)^T] = \mbox{minimum}$$ (2)

If we put (1) into (2) and expand, we get

$$E[\varepsilon \varepsilon^T]= E[ A \theta \theta^T A^T - x \theta^T A^T - A \theta x^T + x x^T]$$ (3)

If we let C_x be the autocorrelation of the field ( E[x x^T] ), $C_\theta$ be the autocorrelation of the observations ( $E[\theta \theta^T]$ ), and $C_{x\theta}$ be the cross correlation between the field and the observations ( $E[x \theta^T]$ ), then we can write the above as

$$C_\varepsilon = A C_\theta A^T - C_{x \theta} A^T - A C_{x \theta}^T + C_x$$ (4)

The next step requires the application of the following matrix identity (proved in the appendix),

(A - B C^-1) C (A - B C^-1)^T - B C^-1 B^T = A C A^T - B A^T - (B A^T)^T (5)

using A in (4) for A in (5), and $C_{x\theta}$ for B as well as $C_\theta$ for C, we can reduce (x) to

$$C_\varepsilon = (A - C_{x \theta} C_\theta^{-1}) C_\theta (A... ...\theta^{-1})^T - C_{x\theta}C_\theta^{-1}C_{x \theta}^T + C_x$$ (6)

(note we have also used the fact that $C_\theta = C_\theta^T$).

The matrices $C_\theta$, is an autocorrelation matrix therefore both it and $C_\theta^{-1}$ are nonnegative definite (see appendix), therefore

$$(A - C_{x \theta} C_\theta^{-1}) C_\theta (A - C_{x \theta} C_\theta^{-1})^T$$ (7)

and

$$C_{x\theta}C_\theta^{-1}C_{x \theta}^T$$ (8)

are both matrices with positive diagonal elements. This means that the diagonal elements of $C_\varepsilon$ are therefore minimized when it is true that,

$$A - C_{x\theta} C_\theta^{-1} = 0$$ (9)

Therefore we have,

$$A = C_{x \theta} C_{\theta}^{-1}$$ (10)

This is the estimator that we are seeking.

$$\hat{x} = C_{x \theta} C_{\theta}^{-1} \theta$$ (11)

Further, we can write down what the expected error for the estimator as,

$$C_\varepsilon = C_x - C_{x\theta} C_\theta^{-1} C_{x \theta}^T$$ (12)

Equations (11) and (12) constitute the Gauss-Markov estimator for the linear minimum means square estimate of a random variable.