Perturbation Growth Test Procedure

Introduction

It is often of interest to validate that the difference between two model runs is due only to machine epsilon (i.e., roundoff) size differences that could have been introduced by, for example,

running the same code on different machine architectures,
running the same code on the same machine architectures, but with different compilation or runtime environments, or
running two code versions that have only structural or perhaps order of operation differences on the same machine.

The nature of global atmospheric models such as CAM is that small solution differences grow rapidly, and hence can hide small errors due to coding, computer system, or porting. The perturbation growth test described here [Ros97] was designed as an inexpensive test to give reasonable confidence that solution differences are due only to rounding differences. There are, however, two important points to keep in mind.

A successful test is not sufficient to guarantee that model solution differences are due only to roundoff differences. Reasons for this include the following.
- Some parameterizations in CAM are extremely sensitive to small perturbations, and in order to have CAM produce simulations that exhibit "acceptable" perturbation growth, it must be run in a special configuration in which some code is disabled via a CPP macro (PERGRO).
- The CLM is extremely sensitive to small perturbations and so the tests are done in "aqua-planet" mode which eliminates interaction with the land model.
- The length of the test is only two days which will only produce a subset of possible atmospheric states, and can't exercise all the code switches that control things like reading boundary datasets.
Hence, not all of CAM's production code is exercised by this test.
Models producing simulations that are the "same to roundoff" (defined as simulations that pass this test) are not guaranteed to produce the same climate. Validating a model's climate still requires that climate simulations and diagnostics be performed.

In spite of the preceding disclaimers, the perturbation growth test is a valuable tool because:

it reliably points out when there is a problem that is making larger than roundoff size changes to the solution, and
it verifies when a change in the solution is characteristic of the change we expect from roundoff level differences.

Test Description

To perform the test requires three model simulations, each two days in duration. The first two are done with the control (trusted) model and differ due to a perturbation introduced into the lowest-order bits of the initial values of the temperature field of one of the runs. The difference between these simulations, as measured by the time evolution of the RMS temperature difference, is used to characterize the perturbation growth of the control model. The third run is done with the model being tested. The difference between the test simulation and either of the simulations from the control model characterizes the growth of the difference between the models.

We say that the test simulation is "within roundoff" of the control simulation if the difference growth does not substantially exceed the perturbation growth. Unfortunately, we can't be more precise than "substantially exceed". After gaining alot of practical experience doing tests of this type, we often find that the difference and perturbation growth curves will lie nearly on top of one another with the perturbation curve lying above the difference curve at times, and lying below it at other times.

The main reason we find this test so useful is that when it fails it generally fails spectacularly, i.e., the difference curve will exceed the perturbation curve by many orders of magnitude within a few model timesteps. On the other hand, a "passed" test may be fuzzy, for example, when the difference curve is consistently above the perturbation curve, but never exceeds it by more than an order of magnitude. As mentioned previously, at this point it's time to perform a full blown climate validation.

Procedure Details

Below the procedure to perform a perturbation growth test is described in detail. The example illustrates the test for the T42 Eulerian dycore, but any dycore, resolution, or constituent scenario could be used.

Configure CAM - To build CAM for perturbation growth testing the configure command must be invoked with the -pergro option (in addition to any options that are specific to the model configuration being tested).
```
% configure -pergro
```
Run CAM - Three two-day simulations are required:
- Run the control model with no perturbation applied to the initial conditions.
- Run the control model with a perturbation applied to the initial temperature field.
- Run the model being tested with no perturbation applied to the initial conditions.
The namelist for a two day run with no perturbation applied to the initial conditions is set up as follows:
```
% build-namelist -case p0 \
    -namelist "&camexp nsrest=0 nelapse=-2 mss_irt=0 nrefrq=0 \
       nhtfrq=1 ndens=1 mfilt=145 hfilename_spec='h%t.nc' \
       empty_htapes=.true. fincl1='T:I','PS:I' aqua_planet=.true.  /"
```
This specifies a two-day initial run, in "aqua-planet" mode, with real*8 precision output of just the T and PS fields at every timestep. Additional variables set for convenience specify no archiving to remote storage, no writing a restart file, and renaming the history file to 'h0.nc'. Note that output of the PS field is required for computing the difference statistic which is a mass weighted RMS value.

The namelist for a two day run with a perturbation applied to the initial temperature field is set up just as above, but with the added namelist variable "pertlim=1.e-14".
Compute RMS temperature differences - The following mass weighted RMS temperature differences need to be computed:
- The difference between the two control model runs, one with no perturbation applied and the second with a perturbed initial temperature field. This difference is used to plot the perturbation growth curve.
- The difference between either run with the control model, and the run with the test model. This difference is used to plot the difference growth curve.
Mass weighted RMS differences for all fields in two CAM history files are computed using the tool cprnc which is distributed with CAM (in the models/atm/cam/tools/cprnc directory). The output from cprnc will contain a line formatted as follows for each temperature timesample in the two files:
```
 MASS WEIGHTED RMS T        6.6882E-12
```
A tool can be written in the user's favorite scripting language to strip the RMS values from the cprnc output to make them available for plotting. An example of such a tool, cprncdf, is a perl script which, if given one filename argument, assumes that the file contains output from cprnc and strips the RMS T values from the file. If cprncdf is given two filename arguments it assumes they are CAM history files on which it invokes cprnc and then processes its output. cprncdf writes output which looks like this to stdout:
```
e48pm_lf_aqpgro
e48ph_aqpgro
T
0 4.9115E-12
1 5.8276E-12
2 6.6468E-12
3 5.0961E-12
...
```
The 3 header lines are comprised of the case names from each of the runs and the name of the variable that the RMS values are for. (cprncdf has a command-line option that allows outputting the RMS differences of any variable in the two files.) Then there is a line for each timesample in the files for which the RMS value is not zero. The first value in the line is the timestep number, followed by the RMS value.
Plot RMS temperature differences - The final step is to compare the perturbation and difference growth curves. Any plotting tool with the ability to create semi-log line plots is sufficient. An example of such a tool, cprncplt, is a perl script which reads files in the format generated by cprncdf (one file for each line in the plot) and then invokes a scientific data processing and plotting tool called yorick to do the plotting. yorick must be installed on your system for this script to be used.

References

[Ros97]: Rosinski, J. M., and D. L. Williamson, The accumulation of rounding errors and port validation for global atmospheric models, SIAM J. SCI. COMPUT. 18(2), pp. 552-564, 1997.