Error composition index: ERRCOMP

The error composition test is not a test of agreement but a test of hypothesis on the composition of the error in the comparison between reference and actual data. The score reflects the concordance with a specified acceptable composition of the error. The error (sum of squared differences between pairs) in a time series can result from differences in the means, the variances, or from random factors. The normally desirable situation is that the random drift is the only significant cause of error. Nevertheless, sometimes a cause of error can be known: for example, a remote sensing device can cause a systematic error in the mean because of bad calibration, or the statistical procedure used to gather the data can have a known dependence on the sample size which makes variance errors possible. In this cases the ERRCOMP score provides a way to raise the global score by computing the error components and calculating a concordance index between the relative importance of each one and a specified 'allowed' partitioning. If, for example, it is expected that a bias (systematic error in the mean) can be as high as 10% of the total error term, the score computed with 0.1 bias allowed will be 1 if the bias error does not exceed 10% and the random error is not less than 90%, and will be progressively lower if the partitioning shifts towards larger biases. Compounding this measure with a DBK or THEIL test through appropriate weights will allow to give scores taking error composition into account as well as actual fit scores. In addition, the error composition terms retrieved through the ERRCOMP statistics can be very useful in identifying the sources of error in model behavior.

The computation of the error components is based on the equation for the partitioned mean squared error

$$\frac{\sum_{i} (Y_i - X_i)^2}{n} = (\overline{Y} - \overline{X})^2 + (S_Y - r S_Y)^2 + (1 + r^2) S_X^2$$ (10)

where $Y_i$ are the predicted data, $X_i$ the actual data, $S_Y, S_X$ the respective standard deviations, and $r$ their correlation coefficient. The left-hand side is the total squared error, and the three components on the right pertain respectively to the mean, variance, and random component. Dividing the right-hand side by the total error gives the relative partitioning, expressed by the error terms $E_m$, $E_s$, $E_r$, which sum up to $1$.

The ERRCOMP test has weights for each component, defaulting to 0 for the mean ($W_m$) and variance ($W_s$) errors and 1 for the random ($W_r$) error. The user can change the weights through the parameters. Then, the ERRCOMP score is computed as

$$ERRCOMP = 1 - \left( d(E_m, W_m) + d(E_s, W_s) + d(E_r, W_r) \right)$$ (11)

where $d(x,y)$ is defined as

[IMAGE png] (12)