Sample run scripts are provided that illustrate how to use configure and build-namelist to set up a production run on an IBM-SP, an SGI Origin, or a PC-Linux platform.
Before we present the example run scripts, we first discuss details of the multitasking strategy employed in CAM 3.0. Both shared-memory multitasking (using OpenMP) and distributed memory (using MPI) multitasking is allowed. Hybrid-mode enables both shared-memory (for CPU's within a node) and distributed-memory (between nodes) multitasking.
Shared-memory multitasking requires the run-time specification of the number of processors to use (by setting the environment variable $OMP_NUM_THREADS). Distributed memory multitasking involves the use of the MPI message-passing programming paradigm. The message-passing code has been validated on IBM SP3, SGI (IRIX), and Linux platforms. Different implementations of MPI handle the settings of the number of tasks in different ways.
The sample run scripts are contained in the directory $CAM_ROOT/models/atm/cam/bld/ in the files run-ibm.csh, run-sgi.csh, and run-pc.csh. They are set up to build and run the default CAM configuration, i.e., Eulerian dynamics at T42 spectral resolution (about 2.8 degree grid spacing) with 26 vertical levels, CLM2 land model, and CSIM4 ice model. The following list describes the features of the scripts, and how to customize them.
cpp directives of the form #include, #if defined, etc., are used to enhance portability, and allow for the implementation of distinct blocks of platform-specific code within a single file. Header files, such as misc.h, are included with #include statements within the source code. When gmake is invoked, the C preprocessor includes or excludes blocks of code depending on which cpp tokens have been defined. cpp directives are also used to perform textual substitution for resolution-specific parameters in the code. The format of these cpp tokens follows standard cpp protocol in that they are all uppercase versions of the Fortran variables, which they define. Thus, a code statement like
parameter(plat = PLAT)
will result in the following processed line (for standard T42 resolution).
parameter(plat = 64)
A parallel gmake is achieved in the build scripts by using gmake with the -j option, which specifies the number of jobs (commands) to run simultaneously.
The output from the build is written to the file MAKE.out in the build directory. If a build fails this is the first place to look for information.
[Rebuilding CAM] The scripts are set up to not rebuild CAM if an executable already exists. To use the script in the situation where a rebuild of only the code (and its dependencies) that has been modified is desired, the script must be edited by commenting out the conditional statement that tests for the existence of a cam executable. Also, the configure command must be commented out since rerunning it will update the configuration files (e.g., misc.h) that all the source files depend on, and hence will result in a complete rebuild.
[Creating the namelist]
The script will change into the build directory and issue the
build-namelist command. The reason for the change to the
build directory is that that is the location of the
config_cache.xml file produced by configure which
is used by build-namelist. The namelist is created in a file
called namelist which is written to the run directory.
The case name, run type, and elapsed length of run are set by assigning values to the shell variables case, runtype, and nelapse respectively. Other changes to the namelist may be made by modifying the -namelist argument of the build-namelist command.
Before discussing the namelist variables, a brief summary is given of the types of model runs. An initial run starts the model from an initial conditions dataset. As the model executes, history datasets, restart datasets and initial condition datasets are periodically written. (see "Model Output Datasets") and "Model generated initial condition dataset files" for more details).
In addition to initial simulations, there are two types of continuation runs: restart and branch. A restart run is an exact continuation of a previous simulation from its point of termination. Namelist variables other than NESTEP, NELAPSE, and NSREST should never be modified for restart runs. A branch run, on the other hand, is a new case that uses restart data from a previous simulation to begin the integration. Since a branch run is a new case, the length of the history interval and the output history fields do not have to be the same as in the control simulation. For example, the branching option can be used to output selected fields more frequently than was the case in the control run. Only history file namelist options should be modified for branch runs. If the user desires to modify other options, such as physics control variables, a new initial run should be done using initial datasets generated by the control run. The user should also note that any namelist variable that can be used as a part or all of a UNIX pathname (e.g. CASEID,NREVSN,..) should only contain characters that are acceptable in UNIX path names.
The set of CLMEXP namelist variables that must be specified depends on whether or not a CLM 3.0 surface dataset is available at the desired model resolution. If such a dataset exists, then build-namelist will provide the minimum set CLMEXP of namelist variables that must be specified:
[Running on an IBM] The default parallelization on the IBM is hybrid MPI/OpenMP. The script is set up for NCAR's machine which has 4 processor nodes. One MPI process (task) is assigned to each node and by default the number of threads is set to the number of processors on each node. The number of nodes is set by either the #@node loadleveler variable if running in batch mode, or by the MP_NODES environment variable if running interactively. For the spectral dynamical cores the number of MPI processes is restricted to being a power of 2.
Changing the number of MPI processes per node is done by setting either
the
#@tasks_per_node loadleveler variable if running in batch
mode, or by the
MP_TASKS_PER_NODE environment variable if
running interactively. If more than 1 MPI process is assigned to a node
then the number of threads for each task should be reduced so that the
number of MPI processes times the number of threads per process does not
exceed the processor count on the node. The number of threads per process
can be set by changing the value of the XLSMPOPTS environment
variable. For example the number of threads is set to 2 with the string
"stack=86000000:parthds=2".
[Running on an SGI] The default parallelization on the SGI is pure OpenMP. The script is setup to use the default number of threads which is set to the number of processors on the machine, or to the number available in the queue if running in batch mode. This number can be reduced by uncommenting the setenv command for OMP_NUM_THREADS environment variable and setting it to the desired number. The script also uses some module commands which set up the system environment for NCAR machines. These commands may need to be commented out or modified on non-NCAR platforms.
[Running on a PC] The PC run script is set up to illustrate a pure OpenMP run. Since the default shared memory processing for the PC in the configure utility is none, the -smp option must be supplied to build CAM for an OpenMP run. The number of processes is set by the shell variable nthreads. The PC script does not have a batch queue submission capability.
[Running batch jobs] The IBM and SGI scripts both contain lines that control batch queue submission. On the IBM the loadleveler variable #@class must be set to a valid queue name, and the run script submitted using the llsubmit command. On the SGI the -q option to the qsub command must be set to a valid queue name, and the run script is submitted using the qsub command.