| CESM User's Guide (CESM1.2 Release Series User's Guide) (PDF) | ||
|---|---|---|
| Prev | Chapter 2. Creating and Setting Up A Case | Next |
Like the CESM1.1 series, the CESM1.2 series also has the new capability to run multiple component instances under one model executable. The only caveat to this usage is that if N multiple instances of any one active component is used, then N multiple instances of ALL active components are required. More details are discussed below. The primary motivation for this development was to be able to run an ensemble Kalman-Filter for data assimilation and parameter estimation (e.g. UQ). However, it also provides you with the ability to run a set of experiments within a single CESM executable where each instance can have a different namelist, and have all the output go to one directory.
In the following an F compset will be used as an illustration. Utilizing the multiple instance code involves the following steps:
create the case
> create_newcase -case Fmulti -compset F -res ne30_g16 -mach hopper > cd Fmulti |
Lets assume the following out of the box pe-layout for hopper:
<entry id="NTASKS_ATM" value="128" /> <entry id="NTHRDS_ATM" value="1" /> <entry id="ROOTPE_ATM" value="0" /> <entry id="NINST_ATM" value="1" /> <entry id="NINST_ATM_LAYOUT" value="concurrent" /> <entry id="NTASKS_LND" value="128" /> <entry id="NTHRDS_LND" value="1" /> <entry id="ROOTPE_LND" value="0" /> <entry id="NINST_LND" value="1" /> <entry id="NINST_LND_LAYOUT" value="concurrent" /> <entry id="NTASKS_ICE" value="128" /> <entry id="NTHRDS_ICE" value="1" /> <entry id="ROOTPE_ICE" value="0" /> <entry id="NINST_ICE" value="1" /> <entry id="NINST_ICE_LAYOUT" value="concurrent" /> <entry id="NTASKS_OCN" value="128" /> <entry id="NTHRDS_OCN" value="1" /> <entry id="ROOTPE_OCN" value="0" /> <entry id="NINST_OCN" value="1" /> <entry id="NINST_OCN_LAYOUT" value="concurrent" /> <entry id="NTASKS_GLC" value="128" /> <entry id="NTHRDS_GLC" value="1" /> <entry id="ROOTPE_GLC" value="0" /> <entry id="NINST_GLC" value="1" /> <entry id="NINST_GLC_LAYOUT" value="concurrent" /> <entry id="NTASKS_WAV" value="128" /> <entry id="NTHRDS_WAV" value="1" /> <entry id="ROOTPE_WAV" value="0" /> <entry id="NINST_WAV" value="1" /> <entry id="NINST_WAV_LAYOUT" value="concurrent" /> <entry id="NTASKS_CPL" value="128" /> <entry id="NTHRDS_CPL" value="1" /> <entry id="ROOTPE_CPL" value="0" /> |
For an F compset, only atm, lnd, rof are full prognostic components. The ocn is a prescribed data component, cice is a mixed prescribed/prognostic component (ice-coverage is prescribed) and glc and wav are stub components. Lets say we want to run 2 instances of CAM in this experiment. The current implementation of multi-instances will also require you to run 2 instances of CLM, CICE and RTM. However, you have the flexibility to run either 1 or 2 instances of DOCN (we can ignore glc and wav since they do not do anything in this compset). To run 2 instances of CAM, CLM, CICE, RTM and DOCN, all you need to do is to change NINST_ATM, NINST_LND, NINST_ICE, NINST_ROF and NINST_DOCN above from 1 to 2. This will result in the following env_mach_pes.xml file:
<entry id="NTASKS_ATM" value="128" /> <entry id="NTHRDS_ATM" value="1" /> <entry id="ROOTPE_ATM" value="0" /> <entry id="NINST_ATM" value="2" /> <entry id="NINST_ATM_LAYOUT" value="concurrent" /> <entry id="NTASKS_LND" value="128" /> <entry id="NTHRDS_LND" value="1" /> <entry id="ROOTPE_LND" value="0" /> <entry id="NINST_LND" value="2" /> <entry id="NINST_LND_LAYOUT" value="concurrent" /> <entry id="NTASKS_ICE" value="128" /> <entry id="NTHRDS_ICE" value="1" /> <entry id="ROOTPE_ICE" value="0" /> <entry id="NINST_ICE" value="2" /> <entry id="NINST_ICE_LAYOUT" value="concurrent" /> <entry id="NTASKS_ROF" value="128" /> <entry id="NTHRDS_ROF" value="1" /> <entry id="ROOTPE_ROF" value="0" /> <entry id="NINST_ROF" value="2" /> <entry id="NINST_ROF_LAYOUT" value="concurrent" /> <entry id="NTASKS_OCN" value="128" /> <entry id="NTHRDS_OCN" value="1" /> <entry id="ROOTPE_OCN" value="0" /> <entry id="NINST_OCN" value="2" /> <entry id="NINST_OCN_LAYOUT" value="concurrent" /> <entry id="NTASKS_GLC" value="128" /> <entry id="NTHRDS_GLC" value="1" /> <entry id="ROOTPE_GLC" value="0" /> <entry id="NINST_GLC" value="1" /> <entry id="NINST_GLC_LAYOUT" value="concurrent" /> <entry id="NTASKS_CPL" value="128" /> <entry id="NTHRDS_CPL" value="1" /> <entry id="ROOTPE_CPL" value="0" /> |
As a result of this, you will have 2 instances of CAM, CLM and CICE (prescribed) and RTM, each running concurrently on 64 MPI tasks - and only 1 instance of DOCN.
A separate user_nl_xxx_NNNN file (where NNNN is the
number of the component instances) will be generated
when cesm_setup is called. In particular,
calling cesm_setup with the above env_mach_pes.xml file
will result in the following user_nl_* files in
$CASEROOT
user_nl_cam_0001 user_nl_cam_0002 user_nl_cice_0001 user_nl_cice_0002 user_nl_clm_0001 user_nl_clm_0002 user_nl_cpl user_nl_docn_0001 user_nl_docn_0002 user_nl_rtm_0001 user_nl_rtm_0002 |
and the following *_in_* files and *txt* files in $CASEROOT/CaseDocs:
atm_in_0001 atm_in_0002 docn.streams.txt.prescribed_0001 docn.streams.txt.prescribed_0002 docn_in_0001 docn_in_0002 docn_ocn_in_0001 docn_ocn_in_0002 drv_flds_in drv_in ice_in_0001 ice_in_0002 lnd_in_0001 lnd_in_0002 rof_in_0001 rof_in_0002 |
The namelist for each component instance can be modified
by changing the corresponding user_nl_xxx_NNNN file for that component
instance. Modifying the user_nl_cam_0002 will result in the namelist
changes you put in to be active ONLY for instance 2 of CAM. To change the
DOCN stream txt file instance 0002, you should place a copy of
docn.streams.txt.prescribed_0002 in $CASEROOT
with the name user_docn.streams.txt.prescribed_0002
and modify it accordlingly.
It is also important to stress the following points:
Different component instances can ONLY differ by differences in namelist settings - they are ALL using the same model executable.
Only 1 coupler component is supported in the CESM1.2 series multiple instance implementation.
user_nl_* files once they are created by cesm_setup ARE NOT removed by calling cesm_setup -clean. The same is true for Macros files.
In general, you should run multiple instances concurrently (the default setting in env_mach_pes.xml). The serial setting is only for EXPERT USERS in upcoming development code implementations.