While the basic DSSAT Shell provides users with simple, effective tools for many simulation exercises, applications that potentially involve thousands of runs are best conducted through batch processing or using specialized tools.
Output Formats
The normal CSM output in ASCII (text) format can be parsed using diverse statistical packages, scripting languages or other software tools. In DSSAT4.7, an option was introduced allowing users to output the main output files in comma-separated variate (CSV) format, as described below. Future work is planned to allow for more generic output where users can select variables of interest and access data programmaticly without using intermediate text or CSV output files.
CSV Output
Increasingly, systems modeling involves large simulation experiments where outputs are analyzed using tools such as spreadsheets, scripting languages or statistical analysis. The standard DSSAT file outputs are formatted for tools within DSSAT but are somewhat difficult to use with external tools. To facilitate analysis with other tools, a new option is provided in DSSAT 4.7 that allows users to output files in column-separated variable (CSV) format (INSERT FIG HERE).
The CVS format is available for EVALUATE.OUT and all .OUT files that contain results as time series. Other output files preserve existing formatting. To facilitate subsequent processing, each row of data additionally contains the run number, experiment identifier, treatment, rotation and replication numbers. The CSV output option can be specified using XBuild, the DSSAT experiment file creation tool, by modifying the “Simulation Options” for Outputs. Under the Format pull-down window, choices are given for text (standard, *.OUT) or CSV formatting. The CSV option is fully implemented for CRGRO (22 crops), CSCER (Wheat, Barley), MZCER (Maize), SGCER(Sorghum), MLCER(Millet) crop models.
Because the CSV format does not require processing to convert each line of output to a specific field width and precision, using the CSV option can increase simulation speed by 10% or more. However, the CSV files will typically be 2.1 to 2.4 times larger than the corresponding *.OUT files.
Detailed information on this option is provided here (link to PDF).
Batch Processing Tools
Command line interface
Command line interface
Individual models can be run from the command line a single line command such as:
C:\DSSAT47\DSCSM047.EXE CSCER047 B DSSBatch.v47
Where:
C:\DSSAT47\DSCSM047.EXE gives the path to the main executable
CSCER047 is the specific module
B is a switch, which indicates that in this case the standard batch execution is requested.
DSSBatch.v47 is the control file that gives the location of file-Xs and specifies treatments to be run.
While the command can be entered manually, the more usual approach is to execute this command from other software. The simplest case in the Windows environment is a batch (*.bat) file that contains the one line command plus optional lines to copy output or pause between runs. If desired, a batch file can be written to execute the model multiple times. With additional Windows commands, output files can be renamed or copied to other directories as needed.
The module must be chosen to match the appropriate crop as specified in the profile, DSSATPRO.v47, which is in the root directory of DSSAT47.
The single letter flags control the simulation mode:
B – Batch, for use with conventional single treatment, single season *.crX files, where “cr” represents a DSSAT crop code (e.g., WH).
N – Seasonal, for use with *.SNX files
Q – Sequence, for use with *.SQX files
S – Spatial, for use witn *.GSX files
The DSSBatch.v47 file lists the name and path of file-x(s) to be used and specifies each treatment, replicate and crop rotation to be simulated. The file-x path can be either absolute (“C:\DSSAT47\Wheat”) or relative (“.\” to specify that file-x is in the same folder where the *.v47 is located. Multiple file names and path can be given, The *.v47 file can be given a different name than DSSBatch but must have the .v47 extension. The file is column formatted and the model input routine currently requires exact column positions for the content. The DSSAT shell generates the DSSBatch.v47 each time a file-x and treatments are selected from the main screen, so this is a handy way to create a *.v47 file, which may be edited or renamed.
The CTR argument overrides the Simulation Controls section of a given File-X, allowing users to select multiple simulation options (e.g. for irrigation or nitrogen management or for ET routines) without having to change the base file-X. Further details are given here.
Implementation in Python
CSM can be run from Python using the ‘subprocess’ module. In this way, user-customized Python scripts can be developed to automate adjustments to input files and to collect pertinent data from model output files. The general Python workflow is as follows: 1) open cultivar, soil, weather, or management files and make parameterization changes, 2) run the model using Python’s subprocess package, 3) open model output files in Python to collect pertinent data, and 4) repeat for other simulation scenarios using program control structures in Python. Users should be warned that the Fortran read statements in the CSM expect input data in a specific format, and corruption of the input files through erroneous Python coding can lead to simulation errors. After changing the working directory to the location of the model executable file, run a CSM batch model simulation with the following Python command:
subprocess.check_call(‘DSCSM047.EXE B DSSBatch.v47’, shell=True)
In this example, ‘DSCSM047.EXE’ is the name of the CSM executable file, ‘B’ specifies a batch mode simulation, and ‘DSSBatch.v47’ provides the name of the batch file, which lists the experiment files and treatments to run. The argument “shell=True” permits the passage of the entire command string ‘DSCSM047.EXE B DSSBatch.v47’ to the Python shell. An ‘import subprocess’ statement must appear at the top of the Python script. The CSM can also be run using a ‘system’ call in Python. With an ‘import os’ statement at the top of the script, use the following command to run CSM:
os.system(‘DSCSM047.EXE B DSSBatch.v47’)
Python documentation recommends use of the subprocess module instead of os.system calls, because subprocess provides more powerful facilities for spawning new processes and retrieving their results. Using the ‘f2py’ package, some Fortran programs can be compiled to run directly from a Python command prompt. The current formulation of the CSM code prevents this practice; however, we are now working toward a solution for this capability.
Advanced modelers and Python users may consider using parallel processing to conduct simulations. This can facilitate the use of the CSM on high-performance computers. Parallel processing in Python is accomplished using the ‘multiprocessing’ package. Generally, the workflow involves 1) creating separate working directories to contain files for each parallel process, 2) copying all required model files to each working directory, 3) establishing ‘worker’ processes using commands from Python’s multiprocessing package, 4) developing a Python function that will be run by each worker process (e.g., incorporating the strategy discussed in the prior paragraph), 5) passing pertinent data or instructions to each worker process using pipes, queues, or shared data arrays as discussed in the Python multiprocessing documentation, and 6) collecting and organizing results from each worker process. If used appropriately, the Python multiprocessing package can radically increase simulation efficiency (i.e., simulations per unit time). However, coding a multiprocessing algorithm is a higher-level programming task, and misuse of the Python multiprocessing package can possibly worsen simulation efficiency as compared to single process applications. For more information on this approach, contact Dr. Kelly Thorp (kelly.thorp@ars.usda.gov).
Implementation in R
Quantum GIS tool
Grid-based Modeling Tools
U. of Chicago
IFPRI
USDA ARS
External Simulation Control Files
External simulation control files can be specified at run time to override the simulation controls in a batch run. See explanation here.
