CESM2 Python Tools

General Overview

CESM2 relies on these NCAR developed external python tools:

• PyCECT - verify a machine port
• CESM_postprocessing - post-process CESM model output
• Cylc- model workflow control manager

These tools are available for use by the community via public github repositories, see below for more information.

PyCECT - Port Verification Tool

CESM Ensemble Consistency Test (CESM-ECT) determines whether a new simulation set up (new machine, compiler, etc.) is statistically distinguishable from an accepted ensemble.

The verification tools in the CESM-ECT suite are:

• CAM-ECT - detects issues in CAM, CLM, and possible other components if using B1850 (12 month runs, CESM2.0)
• UF-CAM-ECT - "Ultra-fast" UF-CAM-ECT detects issues in CAM and CLM (9 time step runs CESM2.0/2.1)
• POP-ECT - detects issues in POP, CICE (12 month run, CESM2.1)

The current ensemble summary files available are:

• CAM-ECT - resolution = f09_g17, compset = B1850 (12 month runs, CESM2.0)
• UF-CAM-ECT - resolution = f19_f19_mg17, compset = F2000climo (9 time step runs, CESM2.0/2.1)
• POP-ECT - resolution = T62_g17 compset = G (12 month run CESM2.1)

Examples

Once these runs are finished, you will need to run:

cime/tools/statistical_ensemble_test/addmetadata.sh

To add metadata to the files to be uploaded. The addmetadata.sh script requires that the nco tool ncks be available in your path, on most hpc systems this is available via: module load nco

CAM-ECT and UF-CAM-ECT

./addmetadata.sh --caseroot /glade/scratch/cesm_user/cesm_tag/case.esm_tag.uf.000 --histfile /glade/scratch/cesm_user/cesm_tag/case.cesm_tag.uf.000/run/case.cesm_tag.uf.000.cam.h0.0001-01-01-00000.nc

./addmetadata.sh --caseroot /glade/scratch/cesm_user/cesm_tag/case.esm_tag.uf.001 --histfile /glade/scratch/cesm_user/cesm_tag/case.cesm_tag.uf.001/run/case.cesm_tag.uf.001.cam.h0.0001-01-01-00000.nc

./addmetadata.sh --caseroot /glade/scratch/cesm_user/cesm_tag/case.esm_tag.uf.002 --histfile /glade/scratch/cesm_user/cesm_tag/case.cesm_tag.uf.002/run/case.cesm_tag.uf.002.cam.h0.0001-01-01-00000.nc

POP-ECT

./addmetadata.sh --caseroot /glade/scratch/cesm_user/cesm_tag/case.cesm_tag.000 --histfile /glade/scratch/cesm_user/cesm_tag/case.cesm_tag.000/run/case.cesm_tag.000.pop.h.0001-12.nc

You are now ready to upload the three files for UF-CAM-ECT, or one file for POP-ECT, to be verified and can use our online CESM2 Ensemble Verification tool

If you have a failure, it could indicate a systems configuration issue. For more information please see the PyCECT github repository README or the DiscussCESM message board.

Technical Reference

A.H. Baker, D.M. Hammerling, M.N. Levy, H. Xu, J.M. Dennis, B.E. Eaton, J. Edwards, C. Hannay, S.A. Mickelson, R.B. Neale, D. Nychka, J. Shollenberger, J. Tribbia, M. Vertenstein, and D. Williamson,"A new ensemble-based consistency test for the community earth system model."Geoscientific Model Development, 8, pp. 2829-2840, Doi: 10.5194/gmd-8-2829-2015, 2015

CESM_postprocessing - Parallel Post Processing Tools

Python based parallel post-processing tools are being used for the CESM2 development, tuning and CMIP6 data processing and include:

• Model output diagnostics plots for atmosphere, land, land-ice, ocean, ocean biogeochemistry, river runoff, and sea-ice
• Compressed single variable time series file generation from model output time slice files
• Climate Model Output Rewriter (CMOR) for CMIP6 compliance

Public access to these tools are available here. The repository wiki contains additional instructions on how to use these tools on NCAR machines.

* Support for these tools is currently limited to NCAR machines only

Cylc - Workflow manager

CESM2 is using the Cylc workflow engine as part of the CMIP6 effort as well as other NCAR projects invovling complex workflows of the CESM2 model. The customized NCAR Cylc related scripts are available from the public github repository here.

* Support for these tools is currently limited to NCAR machines only