This section contains specific examples for creating and running various CCSM3 runs. We have selected examples that deal with issues that have most commonly effected CCSM3 users. Note that all examples in this section assume that short-term archiving is enabled (this will occur as a default setting in the mach_env.$MACH script). It is also assumed that the user will build the case directory in /user/.
The following describes the steps necessary to set up a T42_gx1v3 run with fully active model components. The case name for this run will be TestB. We will subsequently use this run as a reference case for setting up branch and hybrid runs in sections 6.2 and 6.3. TestB will start from 1989-01-01 and run for one year.
CCSM3 contains the ability to automatically resubmit itself via the setting of the environment variable $RESUBMIT (see section 4.7). By setting $RESUBMIT to a value greater than 0, the TestB.$MACH.run will resumit itself and decrement $RESUBMIT by 1. Setting $RESUBMIT to 0 (as is done above), will result in a year run that will not resubmit itself.
The following describes the steps necessary to set up a branch (TestB_branch) run starting from reference case TestB at year 1990-01-01 (see section 6.1). The resulting branch run will run for a month and resubmit itself at the end of that run for another month.
In branch runs all model components start from binary restart files. Consequently, certain constraints apply to the reference cases that are used for starting branch runs. In general, the restart files used to start a branch run have to be consistent with the model version and model settings used in the branch run. In particular, restart files produced by older versions of the CCSM (e.g. CCSM2.0) will not be compatible with current versions of the model. Only restart files generated with the released CCSM3.0 code should be used in doing branch and hybrid runs.
Other restart issues to watch out for are the $CSIM_MODE setting (see below), the $IPCC_MODE setting (see below), and the number of CAM constituents. In setting up a branch run, if the atmospheric component is CAM, the number of atm CAM constituents (see the User's Guide to the NCAR Community Atmosphere Model 3.0 (CAM 3.0)) must be kept the same between the reference case and the branch case. In particular, if $IPCC_MODE is set to OFF or RAMP_CO2_ONLY, then CAM will use 3 constituents. Otherwise, for all other values of $IPCC_MODE, CAM will use 11 constituents. Any run containing 3 CAM constituents may be branched from any other run containing 3 CAM constituents. Similarly, any run containing 11 CAM constituents may be branched from any other run containing 11 CAM constituents. As an example, a run with $IPCC_MODE set to 1870_CONTROL may not be branched from a reference case where $IPCC_MODE is set to RAMP_CO2_ONLY.
Finally, when doing a branch run, the run start date is set by the each component's restart (i.e. $RUN_STARTDATE in env_conf is ignored).
$CSIM_MODE has two options; prognostic and oceanmixed_ice. Restart files produced in oceanmixed_ice mode cannot be used to restart CSIM in any other mode. On the other hand, oceanmixed_ice mode can be initiated using other ice restart files if the CSIM oml_ice_sst_init namelist parameter is set to true.
$RAMP_CO2_START_YMD sets the start of the CO2 ramping and should be set if $IPCC_MODE is set to RAMP_CO2_ONLY. If the branch run is from a RAMP_CO2_ONLY run, the start YMD should be set the same as in the baseline run. There are no tests in CCSM to verify continuity, and users can actually set the start $RAMP_CO2_START_YMD to anything they want.
The following describes the steps necessary to set up a hybrid (TestB_hybrid) run starting from initial/restart data from reference case TestB at year 1990-01-01 where the starting date is 1995-01-01. The hybrid branch run will run for a month and resubmit itself at the end of that run for another month.
Note that for hybrid runs, unlike branch runs, the run startdate in env_conf is used.
$RAMP_CO2_START_YMD sets the start date of the CO2 ramping and should be set if $IPCC_MODE is set to $RAMP_CO2_ONLY. If the reference case for the hybrid run is a $RAMP_CO2_ONLY run, the $RAMP_CO2_START_YMD should be set to the same value as in the reference case run. There are no tests in CCSM3.0 to verify continuity, and users can actually set the start $RAMP_CO2_START_YMD to anything they want.
To set up a production run, the user must run create_newcase, modify the $CASEROOT/env_* files as needed, run configure, interactively build the executables and submit the run script. We provide a recommended process below for doing this.
Once a production run has started, there are several things that need to be monitored.
A production run can fail for several reasons. A model component could "blow up", there could be a machine hardware or software failure, or something in the user's environment might cause the failure.
If a run fails, the following process can be used to clean up and diagnose the failure. When a run fails, it normally shuts down the run automatically. However, there are cases where it can continues to resubmit itself until the RESUBMIT flag is zero, then the run stops completely. This means that several run may run after the failure. These will inevitably fail as well. The user will need to clean up extra runs, diagnose the problem, fix the problem, and restart the run.
First, identify the run that originally failed and remove all files created in subsequent run submissions if there are any. Files may need to be removed in $LOGDIR/, $DOUT_S_ROOT/restart.tars and the stdout and stderr files in $CASEROOT/. Next, look through the logs, stdout, and stderr files in the original run that failed. Search for the error strings in those files. See section 9.1 for more information about the log files. If an error can't be found, restart the run from the last good restart tar file (see section 6.7 for details). Set the $RESUBMIT value to zero to prevent run-away run. Run the single case and check whether the model failed at the same time, and diagnose the problem. Once diagnosed, restart the model and continue running as desired.
This case summarizes the steps to restart a production run. This might occur if a run fails, if restarts are corrupt, if a user needs to continue a run from a previous restart dataset, if files are scrubbed, or if a run is restarted after a period where it's been stopped. This step assumes the source code and scripts are available and addresses primarily the restaging of restart files.
The new CCSM3 scripts make if very simple to have more than one person run a production run. The following describes how to do this.
To generate an IPCC configuration, modify the $IPCC_MODE setting in env_conf before running configure. The supported $IPCC_MODE options are
OFF is the default. The RAMP_CO2_ONLY option will be described below. It is not an IPCC configuration. All other options turn on IPCC features in the active components. These features include sulfates, volcanics, and greenhouse gas forcing. The 1870_CONTROL is a constant 1870 spin-up. Changes to namelist, build, and scripts in general are resolved automatically when configure is run. The CAM cam.buildnml_prestage.csh script is modified significantly when setting the $IPCC_MODE to an IPCC scenario.
To generate a ``ramp_co2'' case, set the $IPCC_MODE value in env_conf to RAMP_CO2_ONLY and set the $RAMP_CO2_START_YMD value to an 8 digit yyyymmdd start date. This is the date that the ramping will begin. When running configure, ramp_co2 namelists will be included in the Buildnml_Prestage/cam.buldnml_prestage.csh.
There are additional CAM specific namelists that can be used to set the CO2 ramp slopes and caps. See the (see the User's Guide to the NCAR Community Atmosphere Model 3.0 (CAM 3.0)) for more information. CAM namelist changes should be added manually to the cam.buildnml_prestage.csh script as needed.
Adding support into CCSM3 for a new machine is generally simple and straight forward. There is one file that must be modified, at most five new files that may need to be generated, and a seventh file that might be to be modified. In most all cases, existing files for a supported machine can be used as a starting point for the new machine.
The first file is a script used to verify that a machine name is valid. It is located at $CCSMROOT/scripts/ccsm_utils/Tools/check_machine.
The next 5 files are new scripts to be located in the directory $CCSMROOT/scripts/ccsm_utils/Machines/. These set up a number of the path names, batch commands, and archive commands needed.
There is a naming convention associated with these files. The file suffixes contain the type of machine (ibm, sgi, linux, etc) and then the machine name. For example, the files for the machine bluesky are
Note there is no modules.ibm.bluesky file. It is not needed for bluesky.
The seventh file that might need to be modified is in the $CCSMROOT/models/bld directory. For each machine type there is a Macros files containing the architecture and machine specific build information including special compiler options, include pathnames, etc. For name of the file used for bluesky is Macros.AIX. Machine name specific information may be ifdef'd in this file. These files eliminate the need to modify any of the Makefile's used by CCSM3.
The following process is recommended for creating scripts for a new machine. For demonstration, a concrete example is presented below where the new machine is an IBM power4 named mypower4.
"run the model" section of run.ibm.mypower4 if needed
The *.l_archive file is not absolutely required. Automated long-term archiving of data will simply not be accomplished without it. It's likely that for an initial port, the l_archive is not needed. However, it is recommended for production runs.
There are a number of ways CCSM3 source code can be modified. Source code can be modified directly in in the appropriate $CCSMROOT/ccsm3/models/ component subdirectory. Alternatively, user-modified code can be placed in the appropriate $CASEROOT/SourceMods/ component sub-directory.
The CCSM build process uses a search path to find files and then builds dependencies automatically. The $CASEROOT/SourceMods/src.* directories are always searched first for source files. These directories are provided as empty directories in $CASEROOT/SourceMods/ by default. By using the SourceMods/src.* directories for user-modified code, the user has the ability to use a shared source code area (for the untarred CCSM3 source distribution) but have locally modified code.
To use the $CASEROOT/SourceMods/src.* directory, the user should copy over the relevant files into the SourceMods/src.*/ directory for the required component(s) before the build script is invoked, make sure $SETBLD is TRUE in env_run, then build and run the case.
If the user wishes to use a subset of their own input datasets rather than those provided in $DIN_LOC_ROOT/inputdata/, they should do the following (as an example we assume that the user would like to replace the cam dataset
AerosolMass_V_64x128_clim_c031022.nc with mydataset_64x128_clim_c040909.nc