The new $create_test tool is located in the $CCSMROOT/scripts/ directory and can be used to set up entire CESM test suites, as well as single standalone test cases. To see the list of test cases, and to view the available script options, execute create_test -help or create_test without any arguments. Creating a single test looks like:
> cd $CCSMROOT/scripts > ./create_test -testname ERS.f19_g16.X.yellowstone_intel -testid t01 > cd ERS.f19_g16.X.yellowstone_intel.t01 > ERS.f19_g16.X.yellowstone_intel.t01.test_build ./ERS.f19_g16.X.yellowstone_intel.t01.submit Check your test results. A successful test produces "PASS" as the first word in the file TestStatus |
The above commands set up an exact restart test (ERS) at the 1.9x2.5_gx1v6 resolution using a dead model component set (X) for the machine yellowstone. The testid provides a unique tag for the test in case it needs to be rerun (i.e. using -testid t02).
As an example, to create an entire suite of tests on yellowstone for the 'prebeta' test category, do the following:
> cd $CCSMROOT/scripts
> ./create_test \
-xml_mach yellowstone \
-xml_compiler intel \
-xml_category prebeta \
-testid alpha01a \
-testroot /glade/scratch/$USER/alpha01a
|
The above command will run all of the 'prebeta' tests in the ccsm_utils/Testlistxml/testlist.xml file that are configured for the category prebeta using the Intel compiler on yellowstone. This is almost identical to the development testing that CSEG carries out on various machines. In addition to creating the suite of tests in your chosen test root, $create_test will also copy several helper scripts into the testroot directory. The first will be named cs.status.$testid.$machine. When run, this script will output the current status of all the tests. The second will be named cs.submit.$testid.$machine. This script is automatically run when all the test cases are created, and it builds and submits the suite of tests for you.
Some things to note about CESM tests:
For usage information about the create_test tool, run "create_test -help".
Test results are output in the TestStatus file. The TestStatus.out file provides additional details of the test run, which may aid in debugging failed tests.
At this time, tests are not always easily re-runnable from an existing test directory. Rather than rerun a previous test case, it's recommended to set up a clean test case, (i.e. create one with a new testid)
Tests are built using the .test_build script. This is different from regular production cases which are built using the .build script. Some tests require more than one executable, thus the .test_build script builds all required executables upfront interactively.
The costs of tests vary widely. Some are short and some are long.
If a test fails, see the Section called Debugging Tests That Failfor debugging assistance.
There are -compare, -generate, and -baselineroot options for the create_test tool that support regression testing. These tools allow one to accomplish several goals:
-generate will save log files as well as coupler history NetCDF files in the baselineroot under the current case name. Later tests will compare their coupler history files against these baselines to check for numerical differences.
-compare will compare the current tag's tests against a previous tag's results, again for numerical accuracy.
-baselineroot simply specifies where you would like your baseline files to be stored. By default, the test system will choose the configured baseline root for your machine.
There are extra test options that can be added to the test such as _D, _E, or _P*. These are described in more detail in the create_test -help output.
There is also a new option to create_test, -nlcompareonly. This allows one to create a suite of Smoke Build Namelist tests. These tests aren't compiled or run, the test cases are simply generated. These are useful in that you can create a suite for a previous CESM tag, then compare the current CESM tag's generated namelists against the previous tag. This can potentially spot hard-to-find answer-changing errors, and/or unintended changes in development.
The test status results have the following meaning
| Test Result | Description |
|---|---|
| BFAIL | compare test couldn't find the baseline directory for the testcase |
| BUILD | build succeeded, test not submitted |
| CFAIL | env variable or build error |
| CHECK | manual review of data is required |
| ERROR | test checker failed, test may or may not have passed |
| FAIL | test failed |
| GEN | test has been generated |
| PASS | test passed |
| PEND | test has been submitted |
| RUN | test is currently running OR it hung, timed out, exceeded its allotted walltime, or exited abnormally |
| SFAIL | generation of test failed in scripts |
| TFAIL | test setup error |
| UNDEF | undefined result |
The following tests are available at the time of writing:
| Test | Description |
|---|---|
| SMS | smoke test |
| ERS | exact restart from startup, default 6 days initial + 5 days restart |
| ERB | branch/exact restart test |
| ERH | hybrid/exact restart test |
| ERI | hybrid/branch/exact restart test |
| ERT | exact restart from startup, default 2 months + 1 month |
| SEQ | sequencing bit-for-bit test |
| PEA | single processor testing with mpi and mpi-serial |
| PEM | pe counts mpi bit-for-bit test |
| PET | pe counts mpi/openmp bit-for-bit test |
| CME | compare mct and esmf interfaces test |
| NCK | single vs multi instance validation test |
| SBN | smoke build namelist test |