CESM1.1 has new capability to run multiple component instances under one model executable. The only caveat to this usage is that if N multiple instances of any one active component is used, then N multiple instances of ALL active components are required. More details are discussed below. The primary motivation for this development was to be able to run an ensemble kalman filter for data assimilation and parameter estimation (e.g. UQ). However, it also provides you with the ability to run a set of experiments within a single CESM executable where each instance can have a different namelist, and have all the output go to one directory.
In the following an F compset will be used as an illustration. Utilizing the multiple instance code involves the following steps:
create the case
> create_newcase -case Fmulti -compset F -res ne30_g16 -mach hopper > cd Fmulti |
Lets assume the following out of the box pe-layout for hopper:
<entry id="NTASKS_ATM" value="128" /> <entry id="NTHRDS_ATM" value="1" /> <entry id="ROOTPE_ATM" value="0" /> <entry id="NINST_ATM" value="1" /> <entry id="NINST_ATM_LAYOUT" value="concurrent" /> <entry id="NTASKS_LND" value="128" /> <entry id="NTHRDS_LND" value="1" /> <entry id="ROOTPE_LND" value="0" /> <entry id="NINST_LND" value="1" /> <entry id="NINST_LND_LAYOUT" value="concurrent" /> <entry id="NTASKS_ICE" value="128" /> <entry id="NTHRDS_ICE" value="1" /> <entry id="ROOTPE_ICE" value="0" /> <entry id="NINST_ICE" value="1" /> <entry id="NINST_ICE_LAYOUT" value="concurrent" /> <entry id="NTASKS_OCN" value="128" /> <entry id="NTHRDS_OCN" value="1" /> <entry id="ROOTPE_OCN" value="0" /> <entry id="NINST_OCN" value="1" /> <entry id="NINST_OCN_LAYOUT" value="concurrent" /> <entry id="NTASKS_GLC" value="128" /> <entry id="NTHRDS_GLC" value="1" /> <entry id="ROOTPE_GLC" value="0" /> <entry id="NINST_GLC" value="1" /> <entry id="NINST_GLC_LAYOUT" value="concurrent" /> <entry id="NTASKS_CPL" value="128" /> <entry id="NTHRDS_CPL" value="1" /> <entry id="ROOTPE_CPL" value="0" /> |
For an F compset, only atm, lnd, rof and cice are prognostic components, whereas the ocn is a data components and glc is a stub component. Lets say we want to run 2 instances of CAM in this experiment. The current implementation of multi-instances will also require you to run 2 instances of CLM. However, you have the flexibility to run either 1 or 2 instances of DOCN (we can ignore glc since it does not do anything in this compset). To run 2 instances of CAM, CLM and CICE, all you need to do is to change NINST_ATM, NINST_LND and NINST_ICE above from 1 to 2. This will result in the following env_mach_pes.xml file:
<entry id="NTASKS_ATM" value="128" /> <entry id="NTHRDS_ATM" value="1" /> <entry id="ROOTPE_ATM" value="0" /> <entry id="NINST_ATM" value="2" /> <entry id="NINST_ATM_LAYOUT" value="concurrent" /> <entry id="NTASKS_LND" value="128" /> <entry id="NTHRDS_LND" value="1" /> <entry id="ROOTPE_LND" value="0" /> <entry id="NINST_LND" value="2" /> <entry id="NINST_LND_LAYOUT" value="concurrent" /> <entry id="NTASKS_ICE" value="128" /> <entry id="NTHRDS_ICE" value="1" /> <entry id="ROOTPE_ICE" value="0" /> <entry id="NINST_ICE" value="2" /> <entry id="NINST_ICE_LAYOUT" value="concurrent" /> <entry id="NTASKS_OCN" value="128" /> <entry id="NTHRDS_OCN" value="1" /> <entry id="ROOTPE_OCN" value="0" /> <entry id="NINST_OCN" value="2" /> <entry id="NINST_OCN_LAYOUT" value="concurrent" /> <entry id="NTASKS_GLC" value="128" /> <entry id="NTHRDS_GLC" value="1" /> <entry id="ROOTPE_GLC" value="0" /> <entry id="NINST_GLC" value="1" /> <entry id="NINST_GLC_LAYOUT" value="concurrent" /> <entry id="NTASKS_CPL" value="128" /> <entry id="NTHRDS_CPL" value="1" /> <entry id="ROOTPE_CPL" value="0" /> |
As a result of this, you will have 2 instances of CAM, CLM and CICE (prescribed) and DOCN, each running concurrently on 64 MPI tasks.
A separate user_nl_xxx_NNNN file (where NNNN is the
number of the component instances) will be generated
when cesm_setup is called. In particular,
calling cesm_setup with the above env_mach_pes.xml file
will result in the following user_nl_* files in
$CASEROOT
user_nl_cam_0001 user_nl_cam_0002 user_nl_cice_0001 user_nl_cice_0002 user_nl_clm_0001 user_nl_clm_0002 user_nl_cpl user_nl_docn_0001 user_nl_docn_0002 |
and the following *_in_* files and *txt* files in $CASEROOT/CaseDocs:
atm_in_0001 atm_in_0002 docn.streams.txt.prescribed_0001 docn.streams.txt.prescribed_0002 docn_in_0001 docn_in_0002 docn_ocn_in_0001 docn_ocn_in_0002 drv_flds_in drv_in ice_in_0001 ice_in_0002 lnd_in_0001 lnd_in_0002 |
The namelist for each component instance can be modified
by changing the corresponding user_nl_xxx_NNNN file for that component
instance. Modifying the user_nl_cam_0002 will result in the namelist
changes you put in to be active ONLY for instance 2 of CAM. To change the
DOCN stream txt file instance 0002, you should place a copy of
docn.streams.txt.prescribed_0002 in $CASEROOT
with the name user_docn.streams.txt.prescribed_0002
and modify it accordlingly.
It is also important to stress the following points:
Different component instances can ONLY differ by differences in namelist settings - they are ALL using the same model executable.
Only 1 coupler component is supported in the CESM1.1 multiple instance implementation.
user_nl_* files once they are created by cesm_setup ARE NOT removed by calling cesm_setup -clean. The same is true for Macros files.
In general, you should run multiple instances concurrently (the default setting in env_mach_pes.xml). The serial setting is only for EXPERT USERS in upcoming development code implementations.