3. Nonlinear Regression

A more difficult type of parametric models is the nonlinear regression model. Here we cannot find just one nice and easy algorithm to solve all problems, as in the linear case. We consider the general parametric model

$$Y=m_\beta(X)+\varepsilon,$$

where $m_\beta$ is a nonlinear function depending on the parameter vector
[3] $\beta=(\beta_0,\ldots,\beta_p)$ and $\varepsilon$ a random error term. Examples can be

$$\begin{eqnarray*} Y&=&\beta X^2+\varepsilon\,\,\textrm{or}\\ Y&=&\beta_1X_1^2+\beta_2X_2^4\,\,\textrm{or}\\ Y&=&\beta_1X^{\beta_2}\,. \end{eqnarray*}$$

Considering these three examples we can distinguish two types of nonlinear regression. The first two examples can be referred to as a curvilinear regression model leading to a simple linear regression model or multiple linear regression model by a transformation of the variables. The third can be called a ``real nonlinear model''. This will not be considered here.

We will face the general curvilinear model

$$Y=\beta_0+\sum\limits_{i=1}^p\beta_if_i(X_i)+\varepsilon$$

where at least one of the functions f_i is assumed to be nonlinear. Using transformed variables $\widetilde X_i=f_i(X_i)$ we simply get the linear model

$$Y=\beta_0+\sum\limits_{i=1}^p\beta_i\widetilde X_i+\varepsilon\,.$$

That means that we only have to transform the data set in the same way and estimate the parameter $\beta$ by the linear regression. As an example, we consider a model leading to the a simple linear regression. It is stored in regr4.xpl. The model is given as

$$Y=2X^2+\varepsilon,$$

where $\varepsilon$ is normal distributed. As in the previous sections, we generate a data set accordingly:

  randomize(0)         ; sets a seed for the random generator
  x=2*uniform(20)-1    ; generates 20 on [-1,1] uniform
                       ;    distributed values
  eps=normal(20)       ; generates 20 standard normal distributed
                       ;    values
  y=2*x^2+eps/2        ; creates y

Next we transform the data $\widetilde X=X^2$ and compute the simple linear regression using the quantlet linreg.

  xtrans=x^2           ; transforms x into the variable xtrans
  {beta,bse,bstan,bpval}=linreg(xtrans,y)
                       ; computes the linear regression

As in the linear case, we get the ANOVA and parameter tables.

  A  N  O  V  A          SS      df     MSS     F-test   P-value
  ______________________________________________________________
  Regression            7.781     1     7.781    30.880   0.0000
  Residuals             4.535    18     0.252
  Total Variation      12.316    19     0.648

  Multiple R      = 0.79483
  R^2             = 0.63175
  Adjusted R^2    = 0.61130
  Standard Error  = 0.50196

  PARAMETERS    Beta       SE       StandB      t-test  P-value
  _____________________________________________________________
  b[ 0,]=    -0.1143     0.1785    -0.0000      -0.640   0.7350
  b[ 1,]=     1.7433     0.3137     0.7948       5.557   0.0000

Both tables are interpreted as in the previous sections. The parameter table shows the estimates for $\beta_0$ and $\beta_1$. From its high p-value, we see that the true value of the constant $\beta_0$ is estimated to be zero. The estimated regression function is

$$\widehat Y(x)= 1.7433 \widetilde x = 1.7433 x^2\,.$$

Now we want to visualize this result.

  data=sort(x~y)             ; creates the graphical object for
                             ;    the data
  setmaskp(data,1,11,8)      ; sets the options for the graphical
                             ;    object data
  rx=#(-10:10)/10            ; creates the vector (-1,-0.9,...,1)
  yq=(beta[2]*rx^2)          ; creates vector with the regressed
                             ;    values
  rdata=sort(rx~yq)          ; creates the graphical object for
                             ;    the regressed values
  rdata=setmask(rdata,"reset","line","red","medium")
                             ; sets the options for the graphical
                             ;    object rdata
  rf=sort(rx~2*rx^2)         ; creates the graphical object for
                             ;    the original function
  rf=setmask(rf,"reset","line","green","medium")
                             ; sets the options for the graphical
                             ;    object rf
  nlplot=createdisplay(1,1)  ; creates the display nlplot
  show(nlplot,1,1,data,rdata,rf) ;shows the display nlplot
  setgopt(nlplot,1,1,"title","Nonlinear Regression")
                             ; sets the title of the display

Figure: Nonlinear regression of $Y=2X^2+\varepsilon$ using the variable transformation. regr4.xpl

Looking at this plot, we see that the transformation of the variables leads to a good reproduction of the shape of the true function, even when the errors are in the same range as the function itself. But we should remember that we have to know the functions f_i.

We don not want to consider the case leading to a multiple linear regression because the principle is exactly the same as with our example. We just have to use the methods of Section 2.

	MD*TECH Method and Data Technologies
	http://www.mdtech.de  mdtech@mdtech.de