This section describes how to use the utilities available in the "models/atm/cam/bld" directory to build and run the model. The configure utility provides a flexible way to specify a particular configuration of CAM. By default it will produce the configuration files required to build the standard production version of CAM (currently Eulerian dynamics at T42 spectral resolution with 26 levels). Invoking the gmake command then builds a CAM executable. The build-namelist utility builds a namelist which specifies the run-time details. It looks at the configuration cache file produced by configure to determine the features of the CAM executable, such as the dynamical core and horizontal resolution, that affect the default specifications for namelist variables. Sample run scripts are provided that illustrate how to use configure and build-namelist to set up a production run on an IBM-SP, an SGI Origin, or a PC-Linux platform. The basic execution procedure is:
CAM2.0 may be built to run in one of two modes. It can run as a stand-alone executable where the land model and the ocean component are contained as part of the executable. This will be referred to as stand-alone mode. In this mode, the model's ocean component uses climatological sea surface temperatures instead of interacting with an external ocean model. Alternatively, CAM2.0 can be run as a component in a system of geophysical models (CCSM) in which two-way interaction is enabled through the use of a flux coupler (Bryan et al., 1996). In this mode, the land, ocean, and sea-ice models are run as separate executables that communicate with CAM via the flux coupler. This will be referred to as flux-coupled mode. Separate build and run procedures are required for running in flux-coupled mode see section 2.1.4. . This section will cover running in stand-alone mode only.
configure produces the configuration build files Filepath, misc.h, params.h, preproc.h, and Makefile. In addition, a configuration cache file (config_cache.xml by default) is written which may be used in a subsequent invocation of configure to exactly reproduce the configuration files. The files produced by running configure are written to the directory where CAM will be built, which by default is the directory from which configure is executed, but can be specified to be elsewhere (see the -cam_bld opttion).
configure will optionally perform tests to validate that the Fortran compiler is operational and Fortran 90 compliant, and that the linker can resolve references to required external libraries (NetCDF and possibly MPI). These tests will point out problems with the user environment in a way that is much easier to understand than looking at the output from a failed build of CAM. We strongly recommend that the first time CAM is built on any new machine, configure should be invoked to execute these tests (see the -test option).
All configuration options can be specified in the following ways, listed in order of decreasing precedence:
At the next level of precedence a few options can be specified by setting environment variables. And finally, at the lowest precedence, many options have hard-coded defaults. Most of these are located in the file config_cache_defaults.xml in the configuration script directory. A few that depend on previous settings are hard-coded in configure (a perl script). The hard-coded defaults will produce the standard production configuration of CAM.
The interactive prompting mode has two levels: basic and expert. The basic mode, which is enabled by the -i option, asks the user all questions required to configure CAM, assuming that the model is built entirely from code that is contained in the distribution. The expert mode, which is enabled by setting the -x option in addition to -i, allows the user to specify that various pieces of code required to build CAM may come from directories outside the distribution. All the flexibility available in the expert interactive mode is also available from the specific options set on the command-line or from a user specified cache file.
The following options may all be specified with either one or two leading dashes, e.g., -help or --help. Options that can be expressed as single letter switches may not be clumped, e.g., -i -x may NOT be expressed as -ix. When multiple options are listed separated by a vertical bar (|), either version may be used.
Table 2.2: Command line arguments to configure
Default: config_cache.xml
Default: ./
Default: directory part of the absolute pathname used to invoke configure
Default: cam
Default: the CAM build directory
Default: config_dir/../../../.. where config_dir is the configuration script directory.
Default: pgcc if using pgf90, otherwise use cc
Default: no debugging
Default: none
Default: eul
Default: cam_bld/esmf where cam_bld is the build directory for CAM.
Default: cam_root/models/utils/esmf where cam_root is the root directory of the CAM distribution.
Default: OS dependent
FFLAGS in the Makefile. If string contains any
whitespace it must be quoted.
Default: ''
Default: The names 'gmake', 'gnumake', and 'make' are checked in order.
In either mode a default value for each setting will be determined based on, in order of decreasing precedence, the specific command-line option for that setting, a user specified default configuration cache file, an environment variable, or a hard-coded default. This default value may then be accepted by entering ``return'', or overridden by the user. Values are checked when possible for legality and consistency.
Default: /usr/local/include except on IBM systems. The IBM Fortran compilers mpxlf90 and mpxlf90_r have the MPI include file location built in.
Default: /usr/local/lib except on IBM systems. The IBM Fortran compilers mpxlf90 and mpxlf90_r have the MPI library location built in.
Default: 1
Default: /usr/local/include
Default: /usr/local/lib
Default: 64
Default: 26
Default: 128
Default: 1
nlat x nlon where nlat
is the number of Gaussian latitudes and nlon is the number of distinct
longitudes. The valid values for spectral dynamics are 64x128,
32x64, or 8x16. For finite-volume dynamics the
meridional grid is equally spaced and includes the pole points. It is specified as
dlatxdlon where dlat is the latitude cell size and dlon is the longitude
cell size, both in
degrees. The valid values for finite-volume dynamics are 2x2.5, 4x5,
or 10x15. The valid resolutions are listed in the
resolution_parameters.xml file in the configuration script directory.
To configure CAM for a resolution that is not in the
resolution_parameters.xml file the value of -res must be set to
custom.
NOTE: some of resolutions recognized by this option are for development purposes only. The recognition of a resolution by this option does NOT imply the existence of a validated control run.
Default: 64x128
Default: SMP is enabled by default only on IBM and SGI systems.
Default: SPMD is enabled by default only on IBM systems.
Default: testing off except in interactive prompting mode
Default: 42
Default: 42
Default: 42
The Filepath file is used by the CAM Makefile to determine which source files will be compiled. The list of source files is comprised of all files with .F90, .F, or .c extensions in each directory listed in Filepath plus the current directory.
The Filepath file is also used by the CAM Makefile to determine which directories will be searched when looking for a source file that can be used to build an object file. The search begins in the current directory, and then proceeds to the directories in the Filepath file, in the order in which they are specified. The first file found will be the one used by make to create the object file.
Default: none
Default: 1
The environment variables recognized by configure are the following:
This section provides a few simple examples of using configure from the command-line and building an executable in interactive prompt mode. In a later section a complete configure, build, and run sequence will be presented in the context of a simple C shell script.
The examples in this section all assume that the CAM configuration script directory is located in the standard place within the CAM distribution. It is also assumed that the CAM configuration directory is not in the user's PATH environment variable. For this reason the configure script must be invoked using an absolute pathname. The script attempts to use the pathname that it was invoked with to determine the configuration script directory. If this is successful then configure can determine the CAM root directory without the user needing to set the CAM_ROOT environment variable.
The following interactive C shell session builds a default production
version of CAM. The shell variable tmpdir is set to a working directory
on the user's system, and camcfg is set to the CAM configuration
directory. We strongly recommend using the -test option the first time
CAM is built on any machine. This will check that the environment is
properly set up so that the Fortran compiler works and can successfully
link to the NetCDF and MPI (if SPMD enabled) libraries.
% cd $tmpdir
% $camcfg/configure -test
configure wrote file: Filepath
configure wrote file: params.h
configure wrote file: misc.h
configure wrote file: preproc.h
configure wrote file: Makefile
configure wrote file: config_cache.xml
Looking for a valid GNU make... using gmake
Testing for Fortran 90 compatible compiler... using mpxlf90_r
Testing NetCDF library... ok
Testing MPI library... ok
configure done.
% gmake -j4 >! make.out
We started by changing into the directory in which the CAM executable will
be built. All the files produced by configure except for the cache file
are required to be in the CAM build directory, so it is generally easiest
to be in that directory when configure is invoked. This example was
carried out on an IBM-SP node. The testing tells us that gmake is a GNU
Make on this machine, that the Fortran compiler is mpxlf90_r, and that
the compiler can successfully reference the NetCDF and MPI libraries. We
then issue the gmake command with a -j4 option, which tells gmake
to use 4 processors for the build.
Output from the make,
except for the messages issued to STDERR, are redirected to the file
make.out. In the event of an error during the build, the make.out
file will contain the command that was issued by gmake that resulted in
the error.
Suppose we type the following command-line:
% $camcfg/configure -dyn fv
** stopping: invalid value of res (64x128) specified as a default
** expected one of: 10x15 2x2.5 4x5
The configure script is telling us that the default resolution of 64x128 is
not valid for the finite-volume dynamics. If we specify a valid resolution
for fv on the command-line:
% $camcfg/configure -dyn fv -res 2x2.5
then configure will successfully generate the output files.
The previous example illustrated that changing a single default value via the command-line options may fail due to dependencies between the configuration parameters. One way to avoid this type of configuration failure is to use the interactive prompting option. The following illustrates such an interactive session:
% $camcfg/configure -i
Entering interactive mode. To accept the default values given in brackets [] just enter return.
Enter directory where CAM will be built [.]: Setting CAM build directory to /ptmp/eaton/bld-test
Enter directory where CAM executable will be created [.]: The CAM executable will be created in /ptmp/eaton/bld-test
Enter dynamics package; eul or sld or fv [eul]: fv Using fv dynamics.
Modify CAM for perturbation growth testing? y or n [n]: NOT configuring CAM for perturbation growth testing.
Choose the finite-volume grid resolution. The valid options are expressed
as dlatxdlon where dlat is the latitude cell size and dlon is the longitude
cell size, both in degrees.
10x15 2x2.5 4x5 [64x128]: 2x2.5
Using horizontal resolution: 2x2.5
Setting number of longitudes to 144 Setting number of latitudes to 91 Setting maximum number of columns in a chunk to 16
Enter number of vertical levels [26]: Setting number of levels to 26
Enter number of advected constituents [1]: Setting number of advected constituents to 1
Enter number of non-advected constituents [1]: Setting number of non-advected constituents to 1
Enter name of CAM executable [cam]: CAM executable will be called cam
Enter Fortran compiler ('default' will use Makefile setting) [default]:
Setting Fortran compiler to default
Enable compiler debugging options? [n]: DISabling compiler debugging options.
Enter Fortran compiler options to be appended to Makefile defaults: []: Setting additional Fortran compiler options ''
Enable SPMD parallelism? [y]: Enabling SPMD parallelism. Enable SMP parallelism (openMP)? [y]: Enabling SMP parallelism. Enable 2D Y-Z parallelism? [n]: Enter directory containing NetCDF include files [/usr/local/include]: Found NetCDF include file in /usr/local/include
Enter directory containing NetCDF library [/usr/local/lib32/r4i4]: Found NetCDF library in /usr/local/lib32/r4i4
Enter directory containing MPI include files []: Enter directory containing MPI library []: creating /ptmp/eaton/bld-test/Filepath creating /ptmp/eaton/bld-test/params.h creating /ptmp/eaton/bld-test/misc.h creating /ptmp/eaton/bld-test/preproc.h creating /ptmp/eaton/bld-test/Makefile creating /ptmp/eaton/bld-test/config_cache.xml
Run tests on the Fortran compiler and external libraries? y or n [y]: Looking for a valid GNU make... using gmake
Testing for Fortran 90 compatible compiler... using mpxlf90_r
Testing NetCDF library... ok
Testing MPI library... ok
configure done.
The only non-default options specified were to use the fv dynamics and to set the horizontal resolution to 2x2.5. In interactive prompt mode configure presents valid resolution names to the user and ensures that a valid value is entered. Note that the default value presented for the location of the NetCDF library was not the hard-coded default (/usr/local/lib). This value came from the LIB_NETCDF environment variable.
build-namelist produces namelists for both CAM and CLM. These are written to a single file (namelist by default) in the directory from which build-namelist is invoked. The only required input for build-namelist is a configuration cache file produced by a previous invocation of configure. The cache file provides information about the configuration of the CAM executable that build-namelist needs to provide the correct default values for things like initial and boundary datasets. The default values are specified in the files DefaultCAMEXPNamelist.xml and DefaultCLMEXPNamelist.xml in the CAM configuration script directory.
The user can specify namelist values that are not set by default, or can override the values that are set by default, in two ways. A file that contains a namelist may be specified using the -infile option, and namelist values may be specified directly on the command-line using the -namelist option. The command-line specification takes precedence over values read from a file, and both take precedence over the default values. In addition a few namelist variables may be set with specific command-line options and these settings take precedence over the values set using -namelist or -infile. There is also an interactive prompting option (-i) which allows the user to view the namelist produced by the default and command-line settings, and make final changes.
The following options may all be specified with either one or two leading dashes, e.g., -help or --help. When multiple options are listed separated by a vertical bar (|), either version may be used.
Table 2.3: Command line arguments to build-namelist
Default: directory part of the absolute pathname used to invoke build-namelist
Default: camrun
Default: config_cache.xml
Default: /fs/cgd/csm/inputdata. This value is set in the files DefaultCAMEXPNamelist.xml and DefaultCLMEXPNamelist.xml in the CAM configuration script directory.
Default: none
-namelist "&camexp nelapse=-10, empty_htapes=.true. /"
Namelist values set on the command-line take precedence over values read from a file specified with the -infile option.
Default: none
Default: namelist
Default: initial
Default: no checking
Default: 1
The environment variables recognized by build-namelist are the following:
This section provides a simple example of using build-namelist from the command-line. Several additional examples of building namelists will be found in Sec. 2.4 .
Assume the shell variable camcfg is set to the CAM configuration
directory and that configure has been invoked to build a default CAM
executable (Eulerian dynamics at T42). The first time a namelist for a
particular CAM configuration is produced, we recommend using the -test
option which checks whether the initial and boundary datasets exist on a
local filesystem. If they do not then a warning is issued to inform the
user which datasets must be copied to the directory from which CAM will be
run.
The invocation of build-namelist,
% $camcfg/build-namelist -test Write out namelist to: namelist
produces the following default namelist:
&camexp absems_data = '/fs/cgd/csm/inputdata/atm/cam2/rad/abs_ems_factors_fastvx.052001.nc' bndtvo = '/fs/cgd/csm/inputdata/atm/cam2/ozone/noaao3.1990.21999.nc' bndtvs = '/fs/cgd/csm/inputdata/atm/cam2/sst/sst_HadOIBl_bc_64x128_clim_c020411.nc' caseid = 'camrun' iyear_ad = 1950 ncdata = '/fs/cgd/csm/inputdata/atm/cam2/inic/gaus/cami_0000-09-01_64x128_T42_L26_c020514.nc' nelapse = -1 nsrest = 0 / &clmexp finidat = '/fs/cgd/csm/inputdata/lnd/clm2/inidata/cam/clmi_0000-09-01_64x128_T42_c020514.nc' fpftcon = '/fs/cgd/csm/inputdata/lnd/clm2/pftdata/pft-physiology' fsurdat = '/fs/cgd/csm/inputdata/lnd/clm2/srfdata/cam/clms_64x128_T42_c020514.nc' /
build-namelist used the configuration cache file (config_cache.xml)
that was produced by configure to determine the dynamics package, land
model, and resolution of the CAM executable. This information was used to
choose the default initial and boundary datasets. The default run type is
an initial run (nsrest=0), the run length is 1 day beyond the date of
the initial conditions (nelapse=-1), and the valid year for the
calculated orbital parameters is 1950 (iyear_ad=1950).
The absence of warnings from build-namelist indicates that all the initial and boundary datasets were found on the local filesystem.
Before we present the example run scripts, we first discuss details of the multitasking strategy employed in CAM2.0 Both shared-memory multitasking (using OpenMP) and distributed memory (using MPI) multitasking is allowed. Hybrid-mode enables both shared-memory (for CPU's within a node) and distributed-memory (between nodes) multitasking.
Shared-memory multitasking requires the run-time specification of the number of processors to use (by setting the environment variable $OMP_NUM_THREADS). Distributed memory multitasking involves the use of the MPI message-passing programming paradigm. The message-passing code has been validated on IBM SP3, SGI (IRIX), and Linux platforms. Different implementations of MPI handle the settings of the number of tasks in different ways.
The sample run scripts are contained in the directory $CAM_ROOT/models/atm/cam/bld/ in the files
run-ibm.csh, run-sgi.csh, and
run-pc.csh. They are set up to build and run the default CAM
configuration, i.e., Eulerian dynamics at T42 spectral resolution (about 2.8
degree grid spacing) with 26 vertical levels, CLM2 land model, and CSIM4
ice model. The following list describes the features of the scripts, and
how to customize them.
camroot must be set to the root
directory of the CAM distribution. This variable is used to set the
directory of the configuration utilities which is assumed to be
$camroot/models/atm/cam/bld.
CSMDATA must be set to the root
directory of the CAM dataset distribution. The value in the run scripts is
appropriate for runs at NCAR on machines that mount the /fs filesystem.
CSMDATA is used by the build-namelist utility to
specify the locations of initial and boundary datasets used by CAM and CLM2.
wrkdir. The defaults are appropriate for NCAR machines and
are large capacity temporary filesystems. The scripts assume that the CAM
build and run directories will be subdirectories of the work directory.
rundir. By default this is set to a subdirectory of the work
directory which has the same name as the run's case name (which is a
required namelist input). The script will create this directory if it
doesn't exist, and will exit if an error is encountered trying to create
the directory.
blddir. By default this is set to the bld
subdirectory of the run directory. The script will create this directory
if it doesn't exist, and will exit if an error is encountered trying to
create the directory.
configure
utility. The default configuration may be changed by supplying the
appropriate arguments to configure.
cam in the
build directory. If no executable exists then the script changes to the
build directory and issues the configure and
gmake commands. If the GNU make command on your system is not
called "gmake", or if it is not in your PATH, then the gmake
in the script will need to be changed to the appropriate name.
gmake is invoked with the -jn option where
n is the number of simultaneous compilation commands issued.
This number should not be larger than the number of processors available
for the build.
MAKE.out
in the build directory. If a build fails this is the first place to look
for information.
cam executable. Also, the
configure command must be commented out since rerunning it
will update the configuration files (e.g., misc.h) that all the source
files depend on, and hence will result in a complete rebuild.
build-namelist command. The reason for the change to the
build directory is that that is the location of the
config_cache.xml file produced by configure which
is used by build-namelist. The namelist is created in a file
called namelist which is written to the run directory.
case, runtype, and
nelapse respectively. Other changes to the namelist may be
made by modifying the -namelist argument of the
build-namelist command.
#@node loadleveler variable if
running in batch mode, or by the MP_NODES environment variable
if running interactively. For the spectral dynamical cores the number of
MPI processes is restricted to being a power of 2.
#@tasks_per_node loadleveler variable if running in batch
mode, or by the MP_TASKS_PER_NODE environment variable if
running interactively. If more than 1 MPI process is assigned to a node
then the number of threads for each task should be reduced so that the
number of MPI processes times the number of threads per process does not
exceed the processor count on the node. The number of threads per process
can be set by changing the value of the XLSMPOPTS environment
variable. For example the number of threads is set to 2 with the string
"stack=86000000:parthds=2".
OMP_NUM_THREADS environment variable and
setting it to the desired number. The script also uses some
module commands which set up the system environment for NCAR
machines. These commands may need to be commented out or modified on
non-NCAR platforms.
configure utility is
none, the -smp option must be supplied to build CAM for an
OpenMP run. The number of processes is set by the shell variable
nthreads.
The PC script does not have a batch queue submission capability.
#@class must
be set to a valid queue name, and the run script submitted using the
llsubmit command. On the SGI the -q option to
the qsub command must be set to a valid queue name, and the
run script is submitted using the qsub command.
Questions on these pages can be sent to... erik@ucar.edu .
Search page