Climate Variability & Change
Working Group CVCWG

How to set up pacemaker experiments

This page describes how to set up pacemaker experiments using the CESM2 Tropical Pacific Pacemakers as an example. Please note that the source code modifications in Step 5 are only valud for CESM2.1.3. If you wish to run a pacemaker simulation with another version of CESM2 you should be able to insert the source code modifications into the relevant locations for your CESM2 version. However, if you want to run a pacemaker simulation with CESM1, different source code modifications will be needed. Please contact the CVCWG co-chairs or liasons if you need to run with CESM1.

Step 1: generate your nudging mask

The mask will determine where you are relaxing SSTs and where you are not. The mask must be defined on the ocean model grid that you are using. In this example, the mask is defined on the POP gx1v7 grid, which was used for the CESM2 tropical pacific pacemakers. Set the regions you want to nudge fully to 1, the regions you don't want to nudge to zero and regions of the buffer zone to some value in between zero and one. Here you can download an example POP grid definition file and python script that can be used to generate the mask used in the CESM2 tropical Pacific Pacemakers. This can be adapted to mask a region of your choosing.

Step 2: generate your SST forcing dataset

Your SST forcing dataset should consist of 1 file per/month with the naming convention XXX.nc.YYYY.DDD.HH, where YYYY refers to the year, DDD refers to the day of the year and HH refers to the hour of the day and XXX refers to the name of your forcing dataset. These files contains your SST forcing dataset interpolated onto the POP grid. For the case of the CESM2 tropical Pacific pacemakers, this SST forcing dataset is the models climatology from 1880 to 2019, derived from the first 10 (macro) members of the forthcoming CESM2 large ensemble, plus the anomalies from the 1880 to 2019 climatology from ERSSTv5. For those with access to the cheyenne supercomputer, this SST dataset can be found at

/glade/p/cesm/cvwg/proj/pacemaker_restore/input/restore/ModifiedSST_ERSSTv5.

Step 3: Combine your mask and forcing SST dataset in one location

Suppose your SST forcing dataset is located at /my/sst/forcing/data/ and your mask is located at /my/mask/mask.nc. Create a new directory for your case where you will link to both your SST forcing files and your mask. Suppose this directory is located at /ssts/for/my/case. Perform the follwing steps to combine your sst forcing data and mask in the required way

cd /ssts/for/my/case/
ln -s /my/mask/mask.nc ./XXX.nc
ln -s /my/sst/forcing/data/XXX.nc.* .

These steps have resulted in soft links to the mask and the SST forcing data in the same folder and the mask being given a name that corresponds to your SST forcing data name (XXX).

Step 4: Set up your case

Let's assume your CESM tag is located in $CESMROOT and you want to set up your case at the location $CASEROOT. You could set up the equivalent of the CESM2 tropical Pacific pacemakers using the following

cd $CESMROOT/cime/scripts
./create_newcase --case $CASEROOT --compset BHISTcmip6 --res f09_g17 --mach cheyenne --run-unsupported

cd $CASEROOT
./xmlchange RUN_TYPE=hybrid
./xmlchange RUN_STARTDATE="1880-01-01"
./xmlchange RUN_REFDIR=XDIRX
./xmlchange RUN_REFCASE=XCASEX
./case.setup

You'd need to replace XDIRX and XCASEX to point toware the directory and the casename of the restarts you want to initialize from.

Step 5: Incorporate the pacemaker sourcemods

Source code modifications are required to run the pacemaker experiments. Here we provide example source code modifications for CESM2.1.3. You should be able to transfer these modifications into the appropriate places in the CESM2 tag that you are using. It is important that you do this translation and don't simply use the source code modifications provided here, unless you are using CESM2.1.3.

There are four POP fortran files that need to be modified: forcing_coupled.F90; forcing.F90; forcing_sfwf.F90; forcing_shf.90. In addition the pop namelist definitions file (namelist_definition_pop.xml) needs to be modified. Copy each of these files from the source code of your CESM tag into your POP SourceMods directory i.e.,

cp $CESMROOT/components/pop/bld/namelist_files/namelist_definition_pop.xml $CASEROOT/SourceMods/src.pop/
cp $CESMROOT/components/pop/source/forcing_coupled.F90 $CASEROOT/SourceMods/src.pop/
cp $CESMROOT/components/pop/source/forcing.F90 $CASEROOT/SourceMods/src.pop/
cp $CESMROOT/components/pop/source/forcing_sfwf.F90 $CASEROOT/SourceMods/src.pop/
cp $CESMROOT/components/pop/source/forcing_shf.F90 $CASEROOT/SourceMods/src.pop/    

This tar file contains the original copy of each of the above files *-ORIG along with the modified copy for CESM2.1.3. You'll need to compare the original and the modified copy to find the changes that have been made. An easy way to do this is using xxdiff from a linux command line e.g.,

xxdiff forcing.F90 forcing.F90-ORIG

Note that xxdiff needs to be loaded on cheyenne first.

module load xxdiff

You'll then need to translate these changes into the appropriate location in your own file within your SourceMods directory.

Step 6: Setting up the POP namelist

You'll need to add the following into $CESMROOT/user_nl_pop

&forcing_shf_nml
luse_cpl_ifrac = .false.
shf_data_inc = 730.
shf_data_renorm(1) = 1.0
shf_data_type = 'n-hour'
shf_file_fmt = 'nc'
shf_filename = '/ssts/for/my/case/XXX.nc'
shf_formulation = 'alyssa_restoring'
shf_interp_freq = 'every-timestep'
shf_interp_inc = 1.e20
shf_interp_type = 'linear'
shf_restore_tau = 2
shf_strong_restore = 0.0
shf_strong_restore_ms = 92.64
shf_weak_restore = 0.
/

where you'd replace "/ssts/for/my/case/" with whatever directory you have placed your SST forcing files in step 3 and you'd replace XXX with whatever name you have given your SST forcing data.

Step 7: Set up and build your case as normal

Now you can modified your namelists and xml parameters, build and submit your case as normal.