Platforms running with SPMD enabled may have trouble when slaves try to get environment variables. This problem can be related to the UNIX shell that the user is using. We have not seen this problem when using tcsh, however, the problem does occur when using "bash" on the Linux platform with the Lahey compiler. Other combinations of shells or platforms may result in this same problem.
The model has undergone extensive testing on IBM platforms both at NCAR and at several other institutions. One problem discovered is a problem compiling the "history.F90" module when using the more extensive "DEBUG mode" compiler options (this mode is set by setting the environment variable DEBUG to "TRUE") and with the SPMD distributed memory configuration. The compiler will sometimes abort with the error "INTERNAL COMPILER ERROR". This problem is due to a problem with the IBM FORTRAN-90 compiler, and has been reported to IBM. In many cases this problem is inconsistent, so recompiling will sometimes (not always) work. As a work around the user could also compile "history.F90" without the DEBUG option on. This problem was reported to IBM and they have fixed the problem in newer versions of the compiler (or with the appropriate "E-fixes" applied to older compilers).
Certain versions of gmake have been shown to have trouble with date-stamps. The Earth System Modeling Framework (ESMF) library has trouble with these versions and will not build. The version of gmake with the fix is 3.79.1, the bug was introduced somewhere between 3.78.1 and 3.79. So to fix the problem you merely need to update to a newer version.
Other times CAM may fail for no obvious reason or perhaps the error message returned is cryptic or misleading. It has been our experience that the majority of these types of symptoms can be attributed to an incorrect allocation of hardware and/or software resources (e.g. the user sets the value of $OMP_NUM_THREADS to a value inconsistent with the number of physical CPUs per node). Often the configuration for machine resources has default settings which are insufficient. Most often an incorrect setting for the per-thread stack size will cause the model to fail with a segmentation fault, allocation error, or stack pointer error. Usually the default setting for this resource is too low and must be adjusted by setting the appropriate environment variables. Values in the range of 40-70 Mbytes seem to work well on most architectures. As a simple troubleshooting step the user may try adjusting this resource, or the process stack size, for their particular application. Here is a list of suggested runtime resource settings affecting the process and/or thread stack sizes.
limit stacksize unlimited
setenv MP_STACK_SIZE 17000000
limit stack size unlimited
setenv XLSMPOPTS "stack=40000000"
limit stack size unlimited
setenv MP_SLAVE_STACKSIZE 40000000
limit stacksize unlimited
limit stacksize unlimited
setenv MPSTKZ 40000000
In addition to properly configuring machine resources, we've identified the following problems often encountered when building and running CAM on the machines here at NCAR.
Questions on these pages can be sent to... erik@ucar.edu .
Search page