A nonlinear example

In the above example, we could solve for the optimal set of parameters because we could separate out the equations to give one just for $a$ and one just for $b$. In order to do this, the cost function must be linear with respect to the parameters, that is the parameters do not show up as nonlinear functions of each other ( e.g. $a^b$, or within transcendental functions, such a cosine). Sometimes this linearity can be achieved by applying a transformation to the original problem to arrive at a new problem which is linear. For example, consider the problem of estimating the amplitude and phase of a signal at a known frequency,

$$\theta_k = A \sin{ ( \omega t_k + \alpha ) }$$ (9)

where $\omega$ is known, $\theta_k$ and $t_k$ are observed and where $A$ and $\alpha$ are to be estimated.

The simplest cost function would be

$$J(a,b) = \sum_{k=1}^{n} (\mbox{error})^2 = \sum_{k=1}^{n} (\theta_k - A \sin{ ( \omega t_k + \alpha ) })^2$$ (10)

Our equations now give us,

$$\frac{\partial J}{\partial a} = -2 \sum_{k=1}^n \left[ \sin{ (\omega t_k + \alpha) } ( \theta_k - \sin{ (\omega t_k + \alpha) } ) \right] = 0$$

$$\frac{\partial J}{\partial b} = -2 \sum_{k=1}^n A \left[ \cos{ (\omega t_k + \alpha) } ( \theta_k - A \sin{ (\omega t_k + \alpha) } ) \right] = 0$$

These equations are hopelessly intertwined, the $\alpha$ terms cannot be pulled out into an equation for $\alpha$ that is separate from the $A$ in the same way that we achieved (7) and (8). The direct solution of these equations requires an approximation involving successive iterations or some other method.

This problem fortunately has the nice property that it can be transformed into another estimation problem which is linear. If we let,

$$x_k = A \sin{ ( \omega t_k + \alpha ) }$$ (11)

$$y_k = A \omega \cos{ ( \omega t_k + \alpha ) }$$ (12)

then with a little trigonometry, we can write our unknowns $A$ and $\alpha$ as,

$$A = \sqrt{ x^2 + (y/\omega)^2}$$ (13)

$$\alpha = \tan^{-1}{ \omega x / y }$$ (14)

(as long as the amplitude and phase are constant, it does not matter where in our data set we apply (13) and (14), so we drop the indexing subscripts). Now can we find an optimal estimate of $x$ and $y$ ? If so, then we are all set. It turns out that we can. The complicating factor here is that our cost function must now account for the two components $x$ and $y$, while we still are relying on the observations $\theta$.

With lots of messy, tedious, but straightforward algebra we can solve this problem to arrive at:

$$x = \frac{1}{D} \sum_{k=1}^n \sin^2{ \omega t_k} \sum_{k=1}^n \theta_k \cos{\omega t_k}$$ (15)

$$y = - \frac{\omega}{D} \sum_{k=1}^n \sin{\omega t_k} \cos{\omega t_k} \sum_{k=1}^n \theta_k \cos{\omega t_k}$$ (16)

for

$$D = \omega \sum_{k=1}^n \cos^2{\omega t_k} \sum_{k=1}^n \sin^2{\omega t_k}$$

$$- \omega \left[ \sum_{k=1}^n \cos{\omega t_k} \sin{\omega t_k} \right]^2$$

How does one come up with this kind of transformation for a new problem ? There are a few fairly standard transformations that you can use, for instance

$$y = a x^b$$

can be converted to a straight line fit problem by taking the logarithm of both sides. But mostly, finding a suitable transformation is a matter of experiance, persistance and luck.