The CESM1.0 POP2 FAQ
The CESM1.0 POP2 FAQ
Nancy J. Norton
Susan Bates
National Center for Atmospheric Research
February 2012
Contents
- Introduction
- Output-File Questions
- What POP2 output files are produced in a CESM1 run?
- What are the POP2 "tavg" files?
- What's in the standard POP2 tavg output files?
- I want to reduce the size of the POP2 tavg output files by eliminating selected fields. Why do you recommend that I don't do this?
- But what if I really want to reduce the size of the POP2 tavg output files? Is there anything I can do?
- So how do I remove these 3D mixing fields from the tavg output files?
- Why might the ocean model stop during initialization after I've modified the POP2 tavg_contents file in my $CASE/SourceMods/src.pop2 directory?
- Why might the ocean model stop after running one month if I've modified the POP2 output history file list in my $CASE?
- Are the fields in the POP2 tavg output files single or double precision?
- Can I create POP2 tavg output files with double-precision fields?
- Interannual Forcing (IAF) Compset Questions
- What are the interannual forcing (IAF) compsets?
- What are the COREv2 datasets?
- Where can I get the COREv2 data?
- What versions of CESM recognize the IAF compsets?
- But I'm using CESM1.0.3. Can I retrofit my version to support the IAF compsets?
- If I create an IAF case on bluefire, will it run out of the box?
- If I create an IAF case elsewhere, what do I need to do to get it to run?
- Non-convergence and Other Run-Time Questions
- My job stopped running with a POP2 "error in solver" or "solver not converged" message. What should I do?
- How do I decrease the POP2 timestep size?
- How much of a reduction in the POP2 timestep size is reasonable?
- POP2 stops with the error message "ERROR: k.e. > 100." What's the problem?
- Configure- and Build-Time Questions
Introduction
This FAQ, a compilation of common CESM1 POP2 user inquiries, is intended to provide POP2-specific advice to CESM1 POP2 users.
Users are encouraged to be familiar with the terminology and concepts in the CESM1 User's Guide.
Users will find detailed usage information for CESM1 POP2 in the POP User's Guide.
The CESM1 POP2 Output Files
- diagnostics files:
- *.pop.dd* general
- *.pop.do* overflows
- *.pop.dt* transport
- *.pop.dv* velocity
- time-averaged history files:
- *.pop.h* (aka "tavg files")
- restart files:
- *.pop.r.* model variables
- *.pop.rh.* time-averaged history
- *.pop.ro.* overflows
- Create annual averages instead of monthly averages.
- Carefully eliminate some fields.
- KAPPA_ISOP
- KAPPA_THIC
- HOR_DIFF
- PV
- KVMIX
- TPOWER
- KVMIX_M
- VVC
- VDC_T
- VDC_S
- UVEL2
- VVEL2
- WVEL2
- Q
- UET
- UES
- WTT
- WTS
- Compress the tavg files.
- cd $CASE/SourceMods/src.pop2
- cp $CODEROOT/ocn/pop2/input_templates/${OCN_RES}_tavg_contents .
- edit $CASE/SourceMods/src.pop2/${OCN_RES}_tavg_contents, replacing the leading "1" character of each variable with the comment symbol, "#"
- configure and build $CASE, if you haven't already done so
- if you have already configured and built $CASE, just submit your job -- the new job will use the new ${OCN_RES}_tavg_contents to determine which tavg fields are active.
- create_newcase -case $CASE ...
- cd $CASE
- configure -case
- add the following line to Buildconf/pop2.buildexe.csh,
just prior to the gmake command:
set pop2defs = "$pop2defs -DTAVG_R8" - build and run $CASE
- cd $CASE
- add the following line to Buildconf/pop2.buildexe.csh,
just prior to the gmake command:
set pop2defs = "$pop2defs -DTAVG_R8" - mv $EXEROOT/ocn $EXEROOT/ocn.single
- mv $EXERUN/ccsm.exe $EXERUN/ccsm.exe.single
- rebuild and run $CASE
What POP2 output files are produced in a CESM1 run?
A CESM1 CASE configured with the active-ocean model, POP2, will produce restart, history, diagnostic, and log files during the course of its run. These files will appear in your CASE's execution directory, and their filenames follow the CESM output-file naming conventions.
In particular, the standard CESM POP2 model produces the following files. See the Model diagnostics and output section of the CESM POP2 User's guide for information on these files and their configurable options.
What are the POP2 "tavg" files?
The CESM1 POP2 model time-averaged history output files are commonly referred to as "the tavg files." See the Model diagnostics and output:Model output files:Time-averaged history files subsection of the CESM POP2 User's guide for more information on the tavg files.
What's in the standard POP2 tavg output files?
The CESM1 POP2 model is configured to produce, out of the box, a recommended set of ocean-model fields in each time-averaged history file. By default, three sets, or "streams," of tavg files are produced during each run segment: monthly averaged fields, daily averaged fields, and one-time fields.
The one-time fields are generated once, at the beginning of each run segment, and contain a handful of time-invariant 3D fields.
By default, the daily averaged fields are bundled into monthly timeseries files. These files are relatively small, because they contain only a handful of 2D fields. Out of the box, these fields are: Surface Potential Temperature (SST), SST2, Mixed-Layer Depth, and Maximum Mixed-Layer Depth.
The monthly averaged files are the largest files, because they contain many fields, several of which are full 3D fields. For a complete list of the fields contained in these files, see the list in the POP2 User's Guide. Because of the monthly files' large size and monthly frequency, a user may be tempted to try to reduce the size of the POP2 output files by eliminating fields from these files. But see the next question...
I want to reduce the size of the POP2 tavg output files by eliminating selected fields. Why do you recommend that I don't do this?
We do not recommend eliminating fields from the POP2 monthly tavg history files if you are running cases for scientific studies.
Why? Because the fields on the monthly tavg files have been carefully selected to provide a full scientific picture of the ocean-model solutions. These fields are essential for use by the standard CESM ocean-model post-processing diagnostics tools, and if you plan to use these tools on your nonstandard files, they will likely fail because required fields are missing from your tavg output files. If you have eliminated fields from your tavg files and the post-processing tools fail, you are on your own.
Furthermore, the tavg-history file fields are tied to various pre-set POP2 run-time diagnostics settings; removing fields will often result in model shut-down because of inconsistent diagnostics settings and tavg-history-file field availablility. Eliminating fields from the tavg files then becomes a rather frustrating trial-and-error hunt for what "works." We do not recommend this approach.
But what if I really want to reduce the size of the POP2 tavg output files? Is there anything I can do?
We repeat: we do not recommend eliminating fields from the POP2 monthly tavg history files if you are running cases for scientific studies.
But if you insist, there are a few options you might consider. Again, you're on your own should problems arise post-processing.
This can be a useful strategy during the early years of a long spin-up run, because often times you will not be analyzing monthly or seasonal variations in the solutions during the spin-up phase.
Obviously, this is a quick way to reduce your POP2 tavg output file size by a factor of 12. But do consider the scientific rationale for your case before deciding on this option.
If you are not planning to analyze ocean mixing, you might try eliminating some or all of the following fields. Eliminating all of them will reduce the size of each monthly tavg history file by approximately 45%.
Note that the ocean-model diagnostics post-processing package will fail when these variables are not available in the monthly tavg files, and you will be on your own if this happens.
This would entail writing your own post-processing scripts or modifying existing ones. Compressing the ocean history files is a fairly effective way of reducing the file size, because of all of the land points, but note that compression is often a time-consuming process and it may not be practical to do this on your system. We recommend that you try compressing a single tavg file and timing the process before committing to this strategy.
Using gzip to compress a gx1v6 POP2 *pop.h.* file reduces the size by 60%.
However, prior to using these compressed files with the ocean-model post-processing software, you would need to uncompress all of your files, which potentially can take a lot of time. The ocean-model post-processing software will not work on compressed files.
So how do I remove these 3D mixing fields from the tavg output files?
If you understand the consequences of eliminating fields from the tavg output files, you can follow the steps below to comment out and thereby eliminate the 3D mixing fields (listed above) from your tavg output files.
You need to be generally familiar with CESM1 and its terminology for the following instructions to make sense:
Why might the ocean model stop during initialization after I've modified the POP2 tavg_contents file in my $CASE/SourceMods/src.pop2 directory?
You can eliminate any of the fields listed above from your tavg history files, and the POP2 model will run just fine. However, if you decide to eliminate additional fields that are not on the list, the POP2 model might stop during its initialization phase. Why?
The POP2 initial.F90 module contains a handful of subroutines that control the intialization of all POP2 modules, and as a result, it knows about all of the POP2 options and related variables that you've activated for your case. At the end of the final POP2 initialization routine, an inter-module consistency checking subroutine, POP_check, is called. Using the model-wide information available to it, this routine identifies some -- but not necessarily all -- inconsistencies among the various options selected in your case.
If any inconsistencies are detected, the POP_check routine will print messages in the ocn.log and/or ccsm.log files, and if the problem is a fatal error, POP_check will cause the ocean model to shut down, thus preventing the user from running an incorrectly specified case. POP_check is the subroutine that generates errors about missiong UISOP or other fields, and it is the routine that will prevent your case from running beyond the ocean-model initialization phase.
Why might the ocean model stop after running one month if I've modified the POP2 output history file list in my $CASE?
MOC and northward transport errors will only occur after the model has run to the point at which a time-averaged history file is to be generated. Why? Because the variables MOC, N_HEAT, and N_SALT are calculated from other time-averaged quantities, which must be available prior to the MOC, N_HEAT, and N_SALT computations. So if you have chosen to compute MOC, N_HEAT or N_SALT, but you have chosen not to keep a time-averaged variable that is needed in the calculation, the model will fail when it tries to compute MOC, etc.
Presently, there are no checks in place that will detect this particular inconsistency prior to the point at which you try to compute these fields, because of complexities in the way the time-averaging and these particular model diagnostics interact. As some users have noted, this is different from the way some other models handle their variables, which we understand makes it more difficult for new CESM users.
Are the fields in the POP2 tavg output files single or double precision?
By default, all POP2 tavg output fields are created in single precision (netCDF "float").
Can I create POP2 tavg output files with double-precision fields?
By default, all POP2 tavg output fields are created in single precision (netCDF "float"), which is usually sufficient and appropriate for scientific analysis.
Sometimes, however, a user might want to generate double-precision tavg fields (netCDF "double"), perhaps for short debugging runs or other special-purpose situations. If so, and if the doubled file size is not a problem, you can follow these steps to generate the double-precision tavg fields:
Note that if you want to modify an existing case so that it, say, creates double-precision fields for a short period of a run, you will need to rebuild your case after modifying Buildconf/pop2.buildexe.csh. But before you do, you should ensure that the $EXEROOT/ocn directory and $EXERUN/ccsm.exe files have been moved or removed.
Interannual Forcing (IAF) Compsets
- CIAF (active-ocean-only with interannual forcing)
- DIAF (active-ice-only with interannual forcing)
- GIAF (active-ocean-ice with interannual forcing)
- Download your own copy of CESM1.0.4 (or higher versions) from the CESM website (follow the CESM Releases link from the main page ). The location where you download the code is called your $CCSMROOT directory.
- Download the interannual data (COREv2) from the GFDL website (http://data1.gfdl.noaa.gov/nomads/forms/mom4/COREv2.html). You want to download the "un-Corrected" data. When using the CESM, corrections to the data are made within the data atmosphere, so please be sure to get the uncorrected data. To learn more about this data, see the GFDL website.
- Let the model know where the downloaded files from the previous step reside:
- Change all occurrences of the string '/ccsm/ocn/iaf' in the
/$CCSMROOT/models/atm/datm/bld/datm.template.streams.xml
file (there are 10 of them) to match the directory name where you put your data. Do this BEFORE you issue the create_newcase command. This way, any case you create from this $CCSMROOT will find the data. - Let the model know where the
CORE2.t_10.ArcFactor.T62.1997-2004.nc file resides. In the
same file that you previously edited,
$CCSMROOT/models/atm/datm/bld/datm.template.streams.xml,
find the filename and change the directory listed three lines above this (in between theindicators) to match the directory where you put your downloaded data. - Issue the create_newcase command, referencing an IAF compset, from your modified $CODEROOT/scripts directory. Configure, build, and submit your job as usual. If you have followed the above steps, your new IAF cases will be configured to properly run on your machine.
What are the interannual forcing (IAF) compsets?
The interannual forcing (IAF) compsets are those compsets that use the data atmosphere model (datm) to provide CESM1 ocean-only(C), ice-only(D), or ocean-ice(G) cases with forcing from version 2 of the Coordinated Ocean-ice Reference Experiments (COREv2) datasets.
The IAF compsets include
What are the COREv2 datasets?
For additional information on the COREv2 forcing datasets, see http://www.clivar.org/organization/wgomd/resources/core/core-ii
Where can I get the COREv2 data?
If you are running IAF cases on the NCAR machines bluefire or yellowstone, you do not need to download the COREv2 data.
If you are running IAF cases elsewhere, you do. See the full CESM1.0.4 instructions below.
What versions of CESM recognize the IAF compsets?
The IAF compsets are available in the CESM Release Version 1.0.4 (CESM1.0.4) and higher, and will be available in the CESM1.1 release.
CESM1.0.4 was released 21 February 2012, and CESM1.1 is due out sometime mid-2012.
But I'm using CESM1.0.3. Can I retrofit my version to support the IAF compsets?
Yes, you can.
Follow the detailed instructions available to you on the CESM Bulletin Board for retrofitting CESM1.0.3 to support the use of IAF compsets.
If I create an IAF case on bluefire, will it run out of the box?
Yes, as long as you use CESM1.0.4 or higher to create your case.
If I create an IAF case elsewhere, what do I need to do to get it to run?
If you want to create an IAF case on a machine other than bluefire at NCAR, it is easiest to start with a version of CESM that already recognizes the IAF compsets, such as CESM1.0.4 or CESM1.1.
However, it is possible to start with CESM1.0.3; in that case, stop reading these instructions and instead follow the full instructions available to you on the CESM Bulletin Board for retrofitting CESM1.0.3 to support the use of IAF compsets.
The following instructions assume you will be starting from a version of CESM that supports IAF, such as CESM1.0.4 or CESM1.1.
Before you can set up and run your IAF case on a machine other than bluefire, you will need to first download the forcing datasets onto your own disk location, then make modifications to the CESM so that it knows how to locate the data.
You will be downloading a lot of data (~85-90 Gbytes), so make sure your disk is large enough to hold all of the data. Also, this disk must be visible from your case's execution directory.
Here's how you can do this:
Non-convergence and Other Run-Time Questions
My job stopped running with a POP2 "error in solver" or "solver not converged" message. What should I do?
If your case has been running successfully for a period of time, but then the ocean model suddenly fails to converge, you might have encountered a time-stepping instability. If so, you will need to decrease the ocean timestep and restart your case. We recommend you back up at least one month prior to the time of nonconvergence, decrease the ocean timestep by roughly 20%, and then restart your case.
See |Section 5|Running a case|The run|Restarting a run| in the CESM1.0 User's Guide for details on how to restart your run.
How do I decrease the POP2 timestep size?
To decrease the POP2 ocean timestep, you must increase the POP2 namelist variable dt_count in the namelist time_manager_nml.
For example, if the ocean grid in your case is gx1v6, then the default value for dt_count is 23. In order decrease the timestep by roughly 20%, you would increase dt_count to 26. To do this, edit your $CASE/Buildconf/pop2.buildnml.csh script, find dt_count and set dt_count = 26.
You do not need to rebuild the executable in order to change dt_count. Just modify $CASE/Buildconf/pop2.buildnml.csh, and the new value of dt_count will be in effect the next time you submit your job.
How much of a reduction in the POP2 timestep size is reasonable?
Generally speaking, if your run has encountered a CFL-type instability, then decreasing the POP2 timestep by 20% should, under most circumstances, be sufficient to overcome the problem, assuming that you restart your run a month or so prior to the "blowup."
If this approach is successful, then you might consider increasing the ocean timestep back to its original value after you have run about a year or so with the reduced timestep.
If this approach is not successful, then you might find that decreasing the timestep by another 10% or so would be enough to overcome the instability.
However, in most cases, we do not recommend that you reduce your timestep much more than about 30%. If you have reduced your timestep by ~30% and your case still "blows up," it is most likely that something else is wrong with your run.
An exception to this advice is during the spin-up phase of a high-resolution (0.1o) ocean case, when further timestep-size reductions may be necessary to overcome initial adjustments.
POP2 stops with the error message "ERROR: k.e. > 100." What's the problem?
POP2 monitors the kinetic energy diagnostic during a model run. When the kinetic energy grows too large, the model shuts down, as you have discovered in your run.
If your case has been running successfully for some period of time before POP2 detects that K.E. is too large, that usually indicates you've encountered a CFL problem, which can usually be solved by cutting the ocean-model timestep by about 20%.
If, on the other hand, this problem occurs within days, or even steps, after ocean-model initialization, this usually indicates that the forcing coming into POP2 is way too strong, undefined, or otherwise "un-earthlike," which is usually caused by user setup error.
If you are using non-standard initial conditions or forcing in your case, it is most likely that you are forcing the POP2 model unrealistically. If this is your situation, you will need to correct the problems with your initial conditions or forcing before continuing your run.
Configure- and Build-Time Questions
What is the meaning of the "pop decomp not set" build-time error message?
If you create any CESM1 CASE configured with the active ocean model (POP2), use only the default ("out-of-the-box") CESM CASE settings, and then build the CASE on a tested machine, you should not encounter a build-time error.
However, if you decide to customize your CASE by changing the number of PE's assigned to POP2, your build might fail with the "pop decomp not set" error message.
Why? Because the number of processors that can be assigned to POP2 is restricted to a certain set of pre-defined values which depend on the ocean-model grid resolution.
If you want to customize the number of processors assigned to POP2 in your CESM1 CASE, you can do this by adding lines to your CASE/Tools/Template/pop_decomp.xml file. This tool was contributed by a member of the CESM community and is undocumented, but by studying the supported decompostions in this file, you should be able to modify this file to support additional PE/decomposition choices.
Note that not all decompositions make sense for the POP2 model, particularly those that result in subdividing the POP2 grid into very small subdomains, because at some point, the boundary calculations will dominate the interior computations. Also note that our general recommendation is to try to define decompositions that result in subdomains that are square or nearly so.
2012-01-25