next up previous contents
Next: 2.5 Model Input Datasets Up: 2. Building and Running Previous: 2.3 Use Cases   Contents


2.4 Sample Run Scripts

Sample run scripts are provided that illustrate how to use configure and build-namelist to set up a production run on an IBM-SP, an SGI Origin, or a PC-Linux platform.

Before we present the example run scripts, we first discuss details of the multitasking strategy employed in CAM 3.0. Both shared-memory multitasking (using OpenMP) and distributed memory (using MPI) multitasking is allowed. Hybrid-mode enables both shared-memory (for CPU's within a node) and distributed-memory (between nodes) multitasking.

Shared-memory multitasking requires the run-time specification of the number of processors to use (by setting the environment variable $OMP_NUM_THREADS). Distributed memory multitasking involves the use of the MPI message-passing programming paradigm. The message-passing code has been validated on IBM SP3, SGI (IRIX), and Linux platforms. Different implementations of MPI handle the settings of the number of tasks in different ways.

The sample run scripts are contained in the directory $CAM_ROOT/models/atm/cam/bld/ in the files run-ibm.csh, run-sgi.csh, and run-pc.csh. They are set up to build and run the default CAM configuration, i.e., Eulerian dynamics at T42 spectral resolution (about 2.8 degree grid spacing) with 26 vertical levels, CLM2 land model, and CSIM4 ice model. The following list describes the features of the scripts, and how to customize them.

Root directory of CAM source distribution
The shell variable CAM_ROOT must be set to the root directory of the CAM distribution. This variable is used to set the directory of the configuration utilities which is assumed to be $CAM_ROOT/models/atm/cam/bld.

Root directory of CAM dataset distribution
The environment variable CSMDATA must be set to the root directory of the CAM dataset distribution. The value in the run scripts is appropriate for runs at NCAR on machines that mount the /fs filesystem. CSMDATA is used by the build-namelist utility to specify the locations of initial and boundary datasets used by CAM and CLM2.

CAM work directory
The work directory is specified by the shell variable wrkdir. The defaults are appropriate for NCAR machines and are large capacity temporary filesystems. The scripts assume that the CAM build and run directories will be subdirectories of the work directory.

CAM run directory
The run directory is specified by the shell variable rundir. By default this is set to a subdirectory of the work directory which has the same name as the run's case name (which is a required namelist input). The script will create this directory if it doesn't exist, and will exit if an error is encountered trying to create the directory.

CAM build directory
The build directory is specified by the shell variable blddir. By default this is set to the bld subdirectory of the run directory. The script will create this directory if it doesn't exist, and will exit if an error is encountered trying to create the directory.

Configuration of CAM
The configuration of CAM is done by calling the configure utility. The default configuration may be changed by supplying the appropriate arguments to configure.

Building CAM
The script will check for an executable named cam in the build directory. If no executable exists then the script changes to the build directory and issues the configure and gmake commands. If the GNU make command on your system is not called "gmake", or if it is not in your PATH, then the gmake in the script will need to be changed to the appropriate name. gmake is invoked with the -jn option where n is the number of simultaneous compilation commands issued. This number should not be larger than the number of processors available for the build.

2.4.1 Use of C preprocessor tokens

cpp directives of the form #include, #if defined, etc., are used to enhance portability, and allow for the implementation of distinct blocks of platform-specific code within a single file. Header files, such as misc.h, are included with #include statements within the source code. When gmake is invoked, the C preprocessor includes or excludes blocks of code depending on which cpp tokens have been defined. cpp directives are also used to perform textual substitution for resolution-specific parameters in the code. The format of these cpp tokens follows standard cpp protocol in that they are all uppercase versions of the Fortran variables, which they define. Thus, a code statement like

parameter(plat = PLAT)

will result in the following processed line (for standard T42 resolution).

parameter(plat = 64)

2.4.2 Details of the gmake procedure

gmake invokes the utility mkSrcfiles to generate a list of source files (written to the file Srcfiles) using each directory listed in Filepath. gmake then invokes the utility mkDepends to create a dependency file (written to the file Depends) in the CAM build directory. If a file listed as a dependency does not exist in the CAM build directory, gmake searches the directories contained in Filepath, in the order given, for a file with that name. The first file found satisfies the dependency. If user-modified code is to be introduced, Filepath should contain, as the first entry (or entries), the directory containing the user code. User code directories are specified with the -usr_src option to configure.

A parallel gmake is achieved in the build scripts by using gmake with the -j option, which specifies the number of jobs (commands) to run simultaneously.

The output from the build is written to the file MAKE.out in the build directory. If a build fails this is the first place to look for information.

[Rebuilding CAM] The scripts are set up to not rebuild CAM if an executable already exists. To use the script in the situation where a rebuild of only the code (and its dependencies) that has been modified is desired, the script must be edited by commenting out the conditional statement that tests for the existence of a cam executable. Also, the configure command must be commented out since rerunning it will update the configuration files (e.g., misc.h) that all the source files depend on, and hence will result in a complete rebuild.

[Creating the namelist] The script will change into the build directory and issue the
build-namelist command. The reason for the change to the build directory is that that is the location of the config_cache.xml file produced by configure which is used by build-namelist. The namelist is created in a file called namelist which is written to the run directory.

The case name, run type, and elapsed length of run are set by assigning values to the shell variables case, runtype, and nelapse respectively. Other changes to the namelist may be made by modifying the -namelist argument of the build-namelist command.

Before discussing the namelist variables, a brief summary is given of the types of model runs. An initial run starts the model from an initial conditions dataset. As the model executes, history datasets, restart datasets and initial condition datasets are periodically written. (see "Model Output Datasets") and "Model generated initial condition dataset files" for more details).

In addition to initial simulations, there are two types of continuation runs: restart and branch. A restart run is an exact continuation of a previous simulation from its point of termination. Namelist variables other than NESTEP, NELAPSE, and NSREST should never be modified for restart runs. A branch run, on the other hand, is a new case that uses restart data from a previous simulation to begin the integration. Since a branch run is a new case, the length of the history interval and the output history fields do not have to be the same as in the control simulation. For example, the branching option can be used to output selected fields more frequently than was the case in the control run. Only history file namelist options should be modified for branch runs. If the user desires to modify other options, such as physics control variables, a new initial run should be done using initial datasets generated by the control run. The user should also note that any namelist variable that can be used as a part or all of a UNIX pathname (e.g. CASEID,NREVSN,..) should only contain characters that are acceptable in UNIX path names.

The set of CLMEXP namelist variables that must be specified depends on whether or not a CLM 3.0 surface dataset is available at the desired model resolution. If such a dataset exists, then build-namelist will provide the minimum set CLMEXP of namelist variables that must be specified:

If a CLM surface dataset is to be created at run time, then the build-namelist will attempt to provide the following CLM namelist variables: The above datasets define the time invariant surface information, but do not provide initial condition information for the land model. If initial conditions at the specified resolution are available, they will provided by build-namelist via the namelist variable "FINIDAT". NOTE: When the user provides FSURDAT or FINIDAT datasets to the model the surface land mask and orography MUST agree with the land mask and orography of the atmospheric model initial conditions dataset (ncdata). Furthermore, the land informaton must agree between the FINIDAT and FSURDAT datasets as well.

[Running on an IBM] The default parallelization on the IBM is hybrid MPI/OpenMP. The script is set up for NCAR's machine which has 4 processor nodes. One MPI process (task) is assigned to each node and by default the number of threads is set to the number of processors on each node. The number of nodes is set by either the #@node loadleveler variable if running in batch mode, or by the MP_NODES environment variable if running interactively. For the spectral dynamical cores the number of MPI processes is restricted to being a power of 2.

Changing the number of MPI processes per node is done by setting either the
#@tasks_per_node loadleveler variable if running in batch mode, or by the
MP_TASKS_PER_NODE environment variable if running interactively. If more than 1 MPI process is assigned to a node then the number of threads for each task should be reduced so that the number of MPI processes times the number of threads per process does not exceed the processor count on the node. The number of threads per process can be set by changing the value of the XLSMPOPTS environment variable. For example the number of threads is set to 2 with the string "stack=86000000:parthds=2".

[Running on an SGI] The default parallelization on the SGI is pure OpenMP. The script is setup to use the default number of threads which is set to the number of processors on the machine, or to the number available in the queue if running in batch mode. This number can be reduced by uncommenting the setenv command for OMP_NUM_THREADS environment variable and setting it to the desired number. The script also uses some module commands which set up the system environment for NCAR machines. These commands may need to be commented out or modified on non-NCAR platforms.

[Running on a PC] The PC run script is set up to illustrate a pure OpenMP run. Since the default shared memory processing for the PC in the configure utility is none, the -smp option must be supplied to build CAM for an OpenMP run. The number of processes is set by the shell variable nthreads. The PC script does not have a batch queue submission capability.

[Running batch jobs] The IBM and SGI scripts both contain lines that control batch queue submission. On the IBM the loadleveler variable #@class must be set to a valid queue name, and the run script submitted using the llsubmit command. On the SGI the -q option to the qsub command must be set to a valid queue name, and the run script is submitted using the qsub command.

next up previous contents
Next: 2.5 Model Input Datasets Up: 2. Building and Running Previous: 2.3 Use Cases   Contents
Jim McCaa 2004-10-22