10 Post Processing

A completed run of CCSM3.0 will produce a large number of files, each with many variables. Post processing these files presents several unique challenges. Various techniques and tools have been developed to meet these challenges. Most involve software that does accompany the CCSM3 distribution. The user will have to download and install these tools separately. In the following sections, some of these tools will be described and their download location identified. Follow on sections will demonstrate various techniques using these tools.

10.1 Software Overview

10.1.1 The netCDF Operators (NCO)

The netCDF Operators are command line executables specifically designed for multi-file bulk processing. They are free, run on numerous operating systems, and are essential in reducing the number of CCSM files to a manageable level. While not officially supported, they are widely used and easy to learn. The operators and on line user's manual can be found at:

nco.sourceforge.net

Table 7 lists the operators and gives a brief description. The user is directed to the web page listed above for specific details, limitations, and usage.

Table 7: netCDF Operators

Operator Name	Function
ncap	arithmetic processor
ncatted	attribute editor
ncbo	binary operator (add, subtract, multiply, divide)
ncea	ensemble averager
ncecat	ensemble concatinator
ncflint	file interpolator
ncks	kitchen sick (variable extraction)
ncra	record averager
ncrcat	record concatinator
ncrename	renamer
ncwa	weighted averager

Specific examples of these operators exist in sections 10.2.1 and 10.2.2.

10.1.2 The NCAR Command Language (NCL)

NCL is a free, interpreted computer language developed at NCAR. It is highly suited to meteorological and climatological data processing and visualization. It runs on UNIX, AIX, IRIX, LINUX, MACOS X, and Windows under cygwin. It handles netCDF, GRIB, and HDF data formats with ease and is officially supported. The following web site contains a large on line library of graphical and data processing scripts that can be downloaded and applied:

www.cesm.ucar.edu/support/

Questions about the software should be directed to ncl-talk@ucar.edu or to Sylvia Murphy (murphys@ucar.edu). NCL exists on all NCAR supercomputers and CGD systems. It can be also be downloaded at:

ngwww.ucar.edu/ncl/download.html

Questions about the NCL binaries and their installing should be directed to Mary Haley (haley@ucar.edu) of the Scientific Computing Division.

10.1.3 Component Model Processing Suite (CMPS)

CMPS is similar in functionality to the netCDF operators (see section 10.1). They are command line executables that are available on NCAR's supercomputers. Unlike the NCO, CMPS only works with CCSM model data. An on line user's guide is available at:

www.cesm.ucar.edu/support/FH/CMPS/index.shtml.

Table 8 lists the current operators and gives a brief description of their function. CMPS is officially supported. The user is advised to visit the web page above or contact Sylvia Murphy (murphys@ucar.edu) for more information and specific usage.

Table 8: Component Model Processing Suite

Operator Name	Function
add.scp	addition
diff.scp	difference/anomalies
e2d.scp	extract all 2D variables
esvc.scp	extract selected variables and concatenate
have.scp	horizontal averaging
iso.scp	iso-level extraction
iso_interp.scp	interpolation of a variable to an iso surface
mult.scp	multiplication
rmse.scp	RMSE
subext.scp	sub-region extraction
tsext.scp	time series extraction
ttest.scp	t-test
topbot.scp	single level extraction
trans.scp	transects
vert.scp	vertical averaging
volave.scp	volume averaging
vert.scp	vertical averaging
zave.scp	zonal averaging

10.1.4 Diagnostic Packages

Comprehensive diagnostic packages exist for each model component. Not all are publicly available. CCSM3 diagnostic packages are designed to do bulk file processing (e.g. concatenate files and average using the NCO), conduct the necessary data processing, and output NCL graphics in a web format. Table 9 contains a list of the currently available packages, their point of contact, and download location.

Table 9: Publicly available diagnostics

Component	POC	Download location
atm	Mark Stevens (stevens@ucar.edu)	www.cgd.ucar.edu/cms/diagnostics/

10.1.5 Commercial Tools

People who run the CCSM use many commercial tools for post processing. These can include IDL, Fortran, MatLab, Yorick, and Vis5D. These are mentioned for information only. Use of these tools is not supported, and it is up to the user to download, install, and debug their own programs.

10.2 Post Processing Strategies

There is no way this guide can provide a detailed set of instructions for all of the post-processing tasks a typical CCSM3.O run requires. What follows is a short discussion of some of the more common tasks.

10.2.1 Selected Variable File Concatenation

CCSM output consists of many files each containing many variables. Often researchers are only interested in a subset of those variables. One strategy for file management is to extract the variables of interest and concatenate them into a smaller subset of files. The netCDF operators (see section 10.1) are well suited to this task. The following is a code snippet from a c-shell script that uses ncks to extract the selected variables T,U, and V and then uses ncrcat to combine the files together.

#**************************************
# extract variables using ncks
#**************************************
ls *.nc > file.txt                     # get the list of files
set fnlst = `cat file.txt`
foreach x ($fnlst)                     # loop over each file
  ncks -A -h -v T,U,V $x temp_$x       # extract variables to temp file
end
#**************************************
# concatinate variables with ncrcat
#**************************************
ncrcat -A -h temp_*.nc concatinated.nc # concatinate all temp files together
#**************************************

10.2.2 Creating Climatologies

Climatologies can be created using the netCDF operators (see section 10.1) or NCL (see section 10.1.2). The code snippet below demonstrates the use of ncra to create annual averages for a set of years using monthly data for those years. ncra is then used again to average the annual averages to create an annual climatology.

#*********************************************************
#     CALC YEARLY ANNUAL AVERAGES FROM MONTHLY DATA
#*********************************************************
@ yr_cnt      = $yr_strt                         # create year counter
@ yr_end      = $yr_strt + $num_yrs - 1          # put space btwn "-" and "1"
while ( $yr_cnt <= $yr_end )                     # loop over years
  set yr_prnt = `printf "%04d" {$yr_cnt}`        # format(change as required)

  ls ${DIR}${case}_${yr_prnt}*.nc > file.txt     # get list of files
  set files = `cat file.txt`
  ncra -O $files ${DIR}${case}_${yr_prnt}_ann.nc # create annual average
end while

#*********************************************************
#     CREATE CLIMATOLOGICAL ANNUAL AVERAGES
#*********************************************************
ls ${DIR}${case}*ann.nc > file.txt               # get list of ann average
set files = `cat file.txt`
ncra -O $files ${DIR}${case}_ANN_climo.nc        # create climatological avg
#*********************************************************

There is a web page devoted to creating climatologies using NCL

www.ncl.ucar.edu/Applications/climo.shtml

This page demonstrates just a few of the climatological related functions available. For a full list, see

www.ncl.ucar.edu/Document/Functions/climo.shtml

10.2.3 Atmospheric Hybrid to Pressure Level Interpolation

The atmospheric model output is on hybrid coordinates (see the User's Guide to the NCAR Community Atmosphere Model 3.0 (CAM 3.0) for details). Often it is desirable to convert these to pressure coordinates. The following web page demonstrates this conversion using NCL (see section 10.1.2):

www.ncl.ucar.edu/Applications/vert.shtml

10.2.4 POP remapping

A special NCL (see section 10.1) library script (pop_remap.ncl) has been developed for POP remapping. This library script comes with the NCL distribution. In order to remap POP, special POP weight and grid files are necessary. SCRIP, which was developed at Los Alamos National Labs, is used to create these files. NCAR maintains a repository of grid and weight files for model configurations that have been scientifically validated (see section 1.5). If a map or weight file for one of these validated configurations is desired, contact Sylvia Murphy (murphys@ucar.edu ) to have the files placed on an anonymous ftp server. The following web site describes how to use pop_remap.ncl and NCL to create other POP graphics:

www.cesm.ucar.edu/support/CSM_Graphics/pop.shtml