Technical Documentation

Project: Applications of numerical optimum control to regional agro-ecological modeling with respect to nutrient fate

Author: Ralf Seppelt
Date: July 14th 2000


Modifications:

Contents

  1. Disclaimer
  2. Concept
  3. Running the Project
  4. Extensions
  5. Local Optimization GridOpt
  6. Files
  7. Directory Structure in /data/Projects/HUNT0
  8. References
          Appendix

1.    Disclaimer


This is a very preliminary first documentation (be happy that there is one). I hope this documentation answers at least some of the important questions. I am sure after reading this, there will be still a bunch of questions. These questions will probably be answered by R.Seppelt@tu-bs.de.

Added to this documentation is an appendix with some instructions how to run the project focused on phosphorus. This project is called HUNTK1. The steps to run the project are the same. There are just some files, that need to be checked, before the model can be run.

2.    Concept

The basic idea of optimizing land use and related variables (fertilization) makes use of the spatial explicit simulation models integrated in SME. The case studies here use the Hunting Creek Model. For an estimation of optimum land use or habitat maps the following steps are run through
  1. A grid search which calculates maps of performance criterion values, as a part of performance criterion
  2. A local optimization which uses these maps to calculate new land use maps (this step actually does not require SME)
  3. The test, if these results are really optimal, is performed by a Monte Carlo Simulation
  4. These results set up the initial Generation of a Genetic Algorithm Optimization.

3.    Running the Project

The following section describes how the above described steps are performed within the HUNT0 Project. The sections afterward go into detail and describe what happens. The entire project can be controlled using script files in ~/HUNT0/Batch. The following sub sections shortly describe the main scripts.

Step 0: Building the Project

Two steps are necessary to compile the project.
First the model has to be compiled with SME. Use the usual SME commands to compile the Model HNMD in HUNT0, or use make0 script file in ~/HUNT0/Batch
 
 
SME project HUNT0 /data/Projects


SME model HNMD
SME scen huntingcreek or SME scen subshed
SME import GLOB.eqns DIN.eqns DOM.eqns HYD.eqns MAC.eqns
SME build HNMD

There are two predefined scenarios "huntingcreek" and "subshed". The "subshed" scenario bases on a sub watershed of Hunting Creek

As the second step, three utility programs and the library for genetic algorithms have to be compiled. This is done using a main makefile in ~/HUNT0. Calling make in this directory calls the makefile in ~/HUNT0/Prg/ga, ~/HUNT0/Prg/gasme, ~/HUNT0/Prg/gridop. make all and make clean are supported.
 

Step 1: Running Grid Search Script

The systematic search through control space is performed using the run_grid_search script. 
We assume a homogeneous landuse and fertilizer application. Runs for each crop (soybeans, wheat and corn) and each amount of fertilizer application (0 – 150 kg/ha) are started. Further runs for forest and fallow.  The generated maps describe values of the performance criterion (J(z) see goal function).

Usage

run_grid_search <scenario> <run>

<scenario> is the usual SME scenario {huntingcreek, subshed}
<run> specifies an unique identifier, here first chosen by user. It is used to generate subdirectories an parameter files for the following steps. One could name this also a sub-scenario. I used digit's (0,1,2,...) but combinations of letters and digits are allowed (e.g. sol1, igg1).

Operation

Several sub-scripts are called:

 
 
    All the sub-directory structure for this new run are created (generate_run <scenario> <run>). If these directories and files exist, the usual unix-error message appears "cannot create directory, directory exist". Don't worry, it'll work. Files/ directories created are:
           The created directories are still empty after this step. The following step creates files for these directories.
    For each crop (soybean, wheat, corn) and each fertilizer amount (default: 0, 25, 50, 75, 100, 150 kg/ha) simulation runs are started, SME is called
  1. For each further landuse (fallow, forest) simulation runs are started
  2. The resulting Maps are renamed and copied so that they are used as input for local optimization, next step (cp_run <scenario> <run>)
  3. The file grid.dat is generated in ~/HUNT0/DriverOutput. It contains the last row of the goal function protocol file.

Output

Maps in ~/HUNT0/Data/Maps/<scenario><run>

Run duration

Approx. three hours for scenario subshed (Solaris, Sparc 10)

Step 2: Local Optimization

Local Optimization bases on the precalculated scenario maps (previous step).
Using the maps from the previous step, lambda is increased stepwise and the landuse distribution for each lambda (number of cells for each habitat type) is written into file.

Usage

run_grid_opt <scenario> <run>

<scenario> is the usual SME scenario {huntingcreek, subshed}
<run> identifier, sub-scenario.

Operation

This script file calls the program gridopt in ~/HUNT0/Prg/gridopt. It reads a parameter file in this directory named <scenario><run>.par. According to the specifications of this parameter file, the file habitat.dat in ~/HUNT0/Prg/gridopt is generated. For selected lambda values (see) maps of optimum land use and fertilization are generated in ~/HUNT0/Data/Maps/OptMaps/<scenario><run>. These are the input files for the spatial simulation with SME. Additionally these maps are written to ~/HUNT0/Prg/gridopt/OutMaps in ArcView/ ArcInfo format using the default grid-import file extension ".asc".

The habitat.dat file contains the distribution of different land use types as a function of the factor lambda which weights the nutrient outflow out of the region. This distribution is displayed using gnuplot running the script.

Output

Run duration

Less than one minute

Step 3. Run SME on local Local Optimization

In this step we apply the calculated maps of optimum land use and fertilization (generated in previous step) to SME and perform a spatial optimization based on this data.

Usage

run_sme_opt <scenario> <run> <map>

<scenario> is the usual SME scenario {huntingcreek, subshed}
<run> identifier, sub-scenario.
<map> number of map to be used (0..9)

Operation

Well, as described above. The precalculated maps are copied from ~/HUNT0/Data/OptMaps/<scenario><run> to ~/HUNT0/Data, so that SME can read them, and SME is started.

<map> specifies the maps which has to be used. According to the specifications in the parameter file of gridopt  (<scenario><run>.par) that has been used, the output maps (~/HUNT0/Data/OptMaps/<sceanrio><run>/CropData# resp. FertData#) are numbered from 0 to at most 9. This <map>-identifier is therefore associated to a certain weight-value.

Output

Simulation result. See ~/DriverOutput/NOut. This file lists the nutrient outflow out of HuntingCreek on each of the 552 days. Other generated files are GOut, WSOut and Infilter.

Run duration

~ 5 Minutes

Step 4. Running Monte-Carlo-Simulation on Local Optimization

An optimum land use and fertilizer map is used as initial seed for a Monte-Carlo Simulation.

Usage

run_montecarlo <scenario> <run> <map> <p0>

<scenario> is the usual SME scenario {huntingcreek, subshed}
<run> identifier, sub-scenario.
<map> number of map to be used (0..9)
<p0> probability of disturbance (0,0..1,0)

Operation

We have equal command line parameters as in step 3. Additionally <p0> specifies the probability of disturbance. E.g. with a probability of <p0> a grid cell is modified in terms of land use and fertilization amount. Note, if you use p0=1.0 it doesn't matter at all which initial map you use.
The script file calls the program moca in ~/HUNT0/Prg/mc which includes the necessary stochastic. The file mc.dat is written to ~/HUNT0/DriverOutput. It contains the last line of the goal function protocol file generated by the usercode in the SME- model, see below.

Output

Step 5. Running Genetic Algorithm Optimization on Local Optimization

Important

Before starting this procedure, the weighting factor used in the user code of the SME model must be set to the value which was used to generate the local optimization map (step 2)! This needs to be done in ~/HUNT0/UserCode/Ucode.cc at the line
static float lambda_n   = 0.4;
Rebuild the SME model!

Usage

run_ga <scenario> <run> <map>

<scenario> is the usual SME scenario {huntingcreek, subshed}
<run> identifier, sub-scenario.
<map> number of map to be used (0..9)

Operation

The script file calls the program gasme in ~/HUNT0/Prg/gasme. Here also a map of local optimization is used as an initial guess for optimization. The first population is generated by Monte Carlo approach. The progress of convergence is logged to the file ga.dat in ~/HUNT0/DriverOutput. This file contains the last row of the goal function protocol. (???)

After analyzing your results of local optimization by varying the weighting parameter you may now optimize the local map using the performance criteria assessing the entire region.

Output

4.    Extensions to SME

SME Internals

The following changes have been made to UCode.cc in ~/UserCode which effect the run of the model HNMD:

External land use and fertilizer maps, Exchange to Genetic Algorithm

The major modification to the HUNT0 project was made to the the user code files UCode.cc and UCode.h in ~/UserCode/.These modifications cover

SME-Random Generated Landuse Maps

Note: the random landuse map generator in SME is not used anymore (see Monte Carlo program documentation). Exchange is performed using data format of previous section.

SME-Local optimization Maps

Note: The local optimization landuse map generator in SME is not used anymore (see gridopt program documentation). Exchange is performed using data format of previous section.

5.    Local Optimization GridOpt

Compilation

Makefile in ~/HUNT0/Prg/gridopt

Start

gridopt <scenario> <run> (<lambda>)

Function

GridOpt calculates a series of land use and fertilizer maps by variation of the weighting factor lambda in the local performance criteria. The parameter file <scenario><run>.par specifies how lambda is to be modified in terms of the range and steps, and which land use maps are to be written for further use in SME.

Parameter File

The parameter file consists of two parts. First part is a list of lambda values, one number each line, ended by a negative value, maximum of 10 values. For these lambda-values landuse and fertilizer maps will be written for further use in SME. After the negative number the values of the weight lambda are specified for which the landuse distribution is to be calculated. Take a look at the contents of  default.par:
 
 
0.0
1.0
2.2
8.0
18.0
-1
0.01  0.001
0.1   0.01
1.0   0.1
3 0.1
20 1.0
50 5.0
300 10.0
This means, gridopt will write maps CropData0, CropData1 to CropData4 and FertData0, FertData1 to FertData4 into ~/HUNT0/Data/OptMaps/<scenario><run> for the lambda-values 0.0, 1.0, 2.2, 8.0 and 18.0 (first part of file). As an example: CropData2 shows the landuse for the lambda value 2.2.
Lambda is increased stepwise to 0.01 incrementing by 0.001 (firts line after -1), then lambda is increased by 0.01 till 0.1 etc. So the last lines denote stop values and steps of increasing lambda. The file habitat.dat is generated for these lambda values

Input

GridOpt reads precalculated maps of SME (generated in step1). These Maps specify and describe values of the performance criterion, the characteristic function. These SME maps have to be in ARC/View format scaled by 10.000 in HNMD.<scenario>.config. The names of the maps are derived from variable abbreviation and scenario or grid-point id
 
Variable 
(see ~/HUNT0/Config/HNMD.<scenario>.config)		 Abbreviation
<abbrv>
YIELD y
OUT_F_SD sd
OUT_F_SF sf

The coding of the grid-points/scenarios follows:
 
<id> f=0 kg/ha f=25 kg/ha f=50 kg/ha f=75 kg/ha f=100 kg/ha f=150 kg/ha
corn c0 c1 c2 c3 c4 c5
soybean s0 s1 s2 s3 s4 s5
wheat w0 w1 w2 w3 w4 w5
fallow fa




forest fo




Output

GridOpt generates the following output (besides error-messages and core dumps)

 

6.    Files

Protocoll of local optimization habitat.dat

This is the content of the file habitat.dat which is generated by gridopt.
 
column contents
1 lambda
2 number of corn cells
3  number of soybean cells
4 number of wheat cells
5 number of fallow cells
6 number of forest cells
7 total amount of fertilizer applied
8 number of total agricultural cells
9
10
11

Protocoll of SME goal function GoalFunctionProtocol

Simulation run within some of HNMD model generates the file ~/HUNT0/DriverOutput/GoalFunctionProtocol.Lots of stuff is written into this file. The most important part of the file is the last line. From this the assessment of the entire simulation can be read out. For testing purposes the rest of the file may be useful too.
 
column contents
1 time step
2 average yield in region
3 minimum yield
4 maximum yield
5 total Nout accumulated over time
6
7
8
9
10 total amount of fertilizer applied g/m²
11 value of internal performance criteria
12 number of corn cells
13  number of soybean cells
14 number of wheat cells
15 number of fallow cells
16 number of forest cells

 Remark:  Format Changed for Projects using Optimization Library Opt.cc in /UserCode. See Appendix 2

7.    Directory Structure in ~/HUNT0

I added an screen shot of a recent directory structure, just to get an idea what's where. In this section I will describe modified and new files with respect to the HUNT project only.

Batch-Files (~/Batch):

generate_run <scenario> <run>
run_grid_search <sceanrio> <run>
run_loop_wheat <sceanrio>
run_loop_soybean <scenario>
run_loop_corn <sceanrio>
run_fallow <sceanrio>
run_forrest <sceanrio>
cp_scenario <sceanrio>
run_sme_opt <scenario> <run> <map>
run_montecarlo <scenario> <run> <map> <p0>
run_ga <scenario> <run> <map>

Input Data (~/Data)

Files in GA-SME-Exchange format, to be copied to CropData in same directory: Cover entire Region (all except open water, urban, rural) with wheat, soybean, corn, fallow, forest
CropData_Wheat


CropData_SoyB
CropData_Corn
CropData_Fallow
CropData_Forest

Files in GA-SME-Exchange format, to be copied to CropData in same directory: Use specified fertilizer level for entire region (all except open water, urban, rural, forest, fallow) 0, 25, 50, 75, 100, 150 kg/ha. Values may be easily changed using a text editor by automatic find and replace (EMACS ESC-%)
FertData_0


FertData_25
FertData_50
FertData_75
FertData_100
FertData_150

Files in GA-SME-Exchange format, but only the first id (length specification) is used to tell SME to generate either a random generated fertilizer map, or a randomly generated habitat map.
CropData_Random


FertData_Random

Additional Programs (~/prg)

Monte Carlo Simulation (~/prg/mc/moca)

Reads in a initial landuse map and fertilizer map from ~/HUNT0/Data/ filename specified in source code, right at the beginning using a #define instruction. Writes out CropData and FertData to ~/HUNT0/Data according to the GA-SME exchange format.
Also specified in the source code of mc.c right at the beginning: the initial probability of modification.

Genetic Algorithm Optimization (~/prg/gasme/gasme)

Output (~/DriverOutput):

Generated by Grid Search (~/Batch/run_fallow, ~/Batch/run_forest, ~/Batch/run_loop_*, ~/Batch/cp_run)

Maps are written in ~/HUNT0/DriverOutput/Maps. The names of the maps are derived from variable abbreviation and scenario or grid-point id (see above)

Generated by Monte Carlo Simulation (~/prg/mc/moca)

mc.dat
This file contains the last row of the goal function protocol file.

Generated by Genetic Algorithm Optimization (~/prg/gasme/gasme)

ga.dat

Contains also the last row of goal function protocol file.
 

8.    References

  1. Spatial Modelling Environment http://swan.cbl.umces.edu/SME3/
  2. GALib: http://lancet.mit.edu/ga/


Appendix 1

by Kerstin Schulze

After Ralf Seppelt developed the landuse optimization project, I started to run it focused on phosphorus. The HNMD model, described above, is set up for nitrogen.

To run the model for phosphorus the model name was changed to HPMD. The project name is HUNTK1. On this project, the models HNMD and HPMD can be run.
The Output for HNMD in the new HUNTK1 project (see GnuPlot graph in Step 2 "Local Optimization") is a little different from the HUNT0 output. The reason are changes in some hydrology files (SWater.cc, SWData, GWData) that have been made in the meanwhile.

Running the Project

To run the project, you can follow the steps 0-5 described above. In this appendix only the variations to the Steps 0-5 are listed.

Step 0. Building the Project

Before you build the project, the file ~/HUNTK1/UserCode/UCode.cc needs to be checked. Depending on the model the amount of fertilizer applied on the fields and the fertilizer price vary. Check the following lines:
  HNMD HPMD
line 26 #define MAXFERTILIZER 150.0 #define MAXFERTILIZER 100.0
line 35 static float lambda_f = 199.0*1.0e-6 static float lambda_f = 40.0*1.0e-6
line 43 float fert_level[] = {0.0, 25.0, 50.0, 75.0, 100.0, 150.0}; float fert_level[] = {0.0, 20.0, 40.0, 60.0, 80.0, 100.0};

To build the project the following SME commands need to be entered. The make0 script in ~/Batch/ can not be used (or needs to be checked every time you start a new model).

SME project HUNTK1 /data/Projects
SME model HNMD or HPMD
SME scen huntingcreek or SME scen subshed
SME import GLOB.eqns HYD.eqns DIN.eqns or DIP.eqns DOM.eqns MAC.eqns PAR.eqns ET.eqns
SME build

Step 1. Running Grid search script

Check the line 2 in ~/Batch/run_grid_search: SME model HNMD or SME model HPMD.

run_grid_search calls the files run_fallow, run_forest, run_loop_soyb, run_loop_wheat and run_loop_corn. In these runs the predefined fertilizer maps are used. As mentioned above, we need different fertilizer maps for HNMD and HPMD. There are two run_xxx files for each landuse type. One for nitrogen and one for phosphorus. The file that is not used needs to be renamed. I used to add an "_N" or "_P" to the file name (e.g. run_loop_wheat_N).

Step 2. Local Optimization

In the file ~/Prg/gridopt/gridopt.c change the same entries as in the ~/UserCode/UCode.cc file.

Step 3. Run SME on local Local Optimization

Check the model name in ~/Batch/run_sme_opt.

Step 4. Running Monte-Carlo-Simulation on Local Optimization

Check the model name in ~/Prg/mc/moca.c. (Also in ~/moca or is this file generated by ~/moca.c?)

Step 5. Running Genetic Algorithm Optimization on Local Optimization

Check the model name in ~/Prg/gasme/gasme.c (Also in ~/gasme or is this file generated by ~/gasme.c?)

Appendix 2

Optimization Library in Opt.cc

Some projects use a seperate C++-file for the optimization procedures formerly place in /UserCode/UCode.cc. The files are names Opt.cc and Opt.h. This projects also use a modified version of SWater.cc. This modification takes care of small subsheds with no open water cells. Even if there is none open water cell in the region, the file NOut is written properly, e.g. the nutrien outflow out of the watershed mouth is calculated.

With this change in file structure also a change in the format of the output file GoalFunctionProtocol was introduced. This is the currect file format for those projects

Format of GoalFunctionOutput for Opt.cc-using Projects

 
column contents Unit
1 time step d
2 value of regional performance criterion J
3 avg_yield (spatial integral divided by area size) g/m²
4 min_yield of all cells g/m²
5 max_yield of all cells g/m²
6 total amount of fertilizer applied g/m²
7 dayly amoint of nitrogen leaving the watershed mouth g/m²
8 accumulated amount of nitrogen leaving the watershed moud dived by elapes time g/m²
9 accumulated amount of net primary produced biomass g/m²
10 number of corn cells #
11 number of soybean cells #
12 number of wheat cells #
13 number of fallow cells #
14 number of forest cells #

Output Files from SWater.cc

This is a short summary of the file writte by the code of SWater.cc which may be usable for analysis of Results:

/Driveroutput/Nout

column contents unit
1 time d
2 nutrient concentration in watershed mouth cell g/m²
3 nutrient concentration in observation cells (specified in /data/SDWata) if present in study area g/m²
4 outflow of nitrogen out of the region (through watershed mouth) at current day g/m²
5 concentration of nutrient in watershed mouth cell g-N/m² per m³-Water
6 total amout of nitrogen in entire area g/m²

/Driveroutput/WSOut

column contents unit
1 time d
2 head surface water in water shed mouth cell m/m²
3 daily outflow out of region through watershed mouth m/m²