Introduction

The purpose of the Simulation Validation Package (SVP) is to provide a way to simplify, and to some extent automate, the processes of calibration and validation of complex simulation models. The package integrates a comprehensive family of tests of hypotheses about a number of variables whose values are expressed by temporal or spatial series. Hypotheses can be expressed as comparisons with available reference data or as qualitative information about the expected model's behavior and the output of a model. The family of tests includes rough tests to check for minimal feasibility of the data (useful for a first definition of the boundaries in parameter space during calibration of a model) and rigorous statistical tests of the agreement between observed and simulated data in both the time and frequency domain. Although the SVP has been developed for simulation analysis purposes, there is no practical or technical reason why it can't be used for performing quick actual data analysis, although it cannot substitute a full-featured statistical package. Important characteristics of the SVP operation are:

• the possibility to use more than one test for each variable to be checked, each one with its own ``weight'' specifying its relative importance, to obtain a ``score'' for each variable, ranging from 0 to 1;
• the possibility of integrating scores for different variables into a single global score, also ranging from 0 to 1, so that the whole performance of a simulation can be summarized and regions of the parameter space can be explored and ranked according to the global score obtained.

Since different variables can have different importance according to the purposes of the model, it is possible to specify a weight for each variable, so that the experimenter's priorities are taken into account in computing the global score.

In addition to that, all data computed for the tests (basic statistics, indexes, periodograms, regression coefficients, etc.) are available to the experimenter and can be saved to files for later use. The SVP includes a report generation facility which allows complete flexibility in generating human or computer-understandable output.

This guide documents the latest version of the SVP. Although the basic details on the statistical tests are given, the user is urged to refer to the bibliography for authoritative sources on the methods used. The tests in the current version are intended to be used for non-replicated data, as usually output by a single run of a deterministic simulation models. Tests for replicated data may be added in the future. The text that follows uses the words reference data to indicate the observed data or desired outcome, and actual data to indicate the outcome of the model under evaluation. Some of the tests do not require reference data at all, others require data but do not require paired observations, others again require point-by-point matching of reference and actual data along the indexing dimension. The data series are assumed in the writing to be indexed by time, but the same considerations and methods apply to spatially indexed data as well.