Note: The CCSM3.0 POP User's Guide describes POP build procedures in two different contexts: stand-alone POP, via the standard LANL 1.4.3 release, and fully coupled POP, via the CCSM3.0 release. Documentation for both follows, the bulk of which is identical to the standard LANL release. Where they occur, differences between stand-alone and coupled implementation and options are noted.
Although build procedures for the stand-alone CCSM version of POP are not officially supported by CCSM developers, it should be a fairly straightforward process for users to modify the CCSM build procedure and run scripts to build and run the stand-alone code, if they wish to do so.
The procedure for building and running the CCSM POP is fully described in the CCSM3.0 User's Guide, the guide to the fully coupled system. In the following discussion, a basic familiarity with the contents of that guide is assumed, including the CCSM run scripts, the CCSM environment variables, and the CCSM ocean scripts.
The CCSM 3.0 release does not contain the files mentioned in the following paragraphs. Interested users can obtain them from the most recent LANL POP release.
The stand-alone POP make procedure consists of several steps that are governed by options in an architecture specific archdir.gnu file. The correct .gnu file is chosen based on an environment variable called ARCHDIR. The ARCHDIR environment variable is a combination of vendor name and communication paradigm. For example, if you are compiling for an SGI Origin2000 you will have the choice of sgi_mpi, sgi_shmem or sgi_serial depending on whether you want to run in parallel using MPI or SHMEM for the message passing or whether you want to run serially (on a single processor). The archdir.gnu file contains the proper paths and compiler options for the particular architecture you have chosen. If an archdir.gnu file does not exist for your choice, it is often simple to copy a version for another architecture and edit it to set it up for your particular machine. If your site has an unusual setup for locations of compilers and various other tools, you may also have to edit the archdir.gnu file to reflect the different setup.
Once the ARCHDIR environment variable has been set, typing gmake should start the make process. The make process includes a step which runs scripts to generate dependencies for the makefile. After the dependencies are generated, the source code passes through a preprocessor and the resulting routines are finally compiled into an executable named pop(or pop_db if debugging options are chosen, ocn if the executable is generated by the CCSM ocean setup script).
There are a few options for POP that must be determined at compile time. Except for the model size, these are implemented with preprocessor directives which are handled by a C-language preprocessor (cpp) or equivalent (if the Fortran compiler understands such directives).
The full model size must be set in the Fortran module model_size in the file model_size.F. The horizontal extents of the grid imt_global and jmt_global are defined here as well as the number of vertical levels km. Lastly, the number of tracers nt must be defined here and must be at least two to handle potential temperature and salinity which are always assumed to be tracer number one and two, respectively. Several model_size.F files are available with the standard LANL distribution for some standard configurations and the correct version may already have been chosen during the setup of the current run directory.
In the CCSM3.0 POP release, model_size.F files are available for all of the resolutions supported by CCSM. These files reside in the CCSM POP models/ocn/pop/input_templates directory, and are identified by a resolution prefix (eg, gx1v3_model_size.F). The CCSM pop scripts automatically select the appropriate version of the model_size.F file and make it available for the building of the pop executable.
This requirement of setting the model size at compile time will be removed in future versions when dynamic memory is implemented.
The number of processors to be used for a POP simulation must currently be determined at compile time. The parameters NPROC_X and NPROC_Y appear in the archdir.gnu (standard LANL) or models/bld/Macros.$ARCH (CCSM) file and specify the number of processors to assign to the X and Y dimensions. In CCSM, the values for NPROC_X and NPROC_Y are automatically determined within the ocean setup script, based upon the number of processors allocated to the ocean model.
On a standard latitude-longitude grid, the X (or I) and Y (or J) dimensions correspond to longitude and latitude, respectively. For example, to run on a single-processor workstation, you would use
–D NPROC_X=1 –D NPROC_Y=1
The –D flag 'defines' NPROC_X to have the value 1.
Now suppose that you want to run a POP simulation with a 384x288 horizontal grid (the 2/3-degree modified displaced-pole grid) on a 32-processor machine. You might specify that:
–D NPROC_X=8 –D NPROC_Y=4
This will create 8x4=32 sub-domains, each with 384/8=48 points in the X dimension by 288/4=72 points in the Y dimension. If the number of processors in a given dimension does not evenly divide the size of the axis, the code will attempt to pad the domain in the last processor to accomodate the difference. For example, if we were to use the same 384x288 horizontal grid, but wished to use 15 processors (NPROC_X=5 and NPROC_Y=3), 15 subdomains would be created, each of size 77x96. Because 77x5=385, the last processor in the X-direction will only have 76x96 ocean points and one extra row of padding to bring the domain up to 77x96. Although this extra row of padding does not affect the calculation, the code will perform operations on those extra points and such padding should be minimized. In some cases, this may not be possible and the code will attempt to exit with an appropriate error message, although CCSM experience has been that sometimes the model will just "hang" instead.
These preprocessor options will also disappear in later versions when this information can be determined at run time.
The choice of processor number can affect performance. Using too many processors for a small problem can result in more communication of data between processors than actual computation on a processor. Alternatively, using too few processors results in long run times. These and other performance considerations on different classes of computers are discussed in the Installation Manual; and during installation, the code should have been set up for best performance on your system. However, you must decide how many processors to assign to your job through the NPROC_X and NPROC_Y parameters. Therefore, you should be aware of how the number (NPROC_X * NPROC_Y) and layout (NPROC_X,NPROC_Y) affects the performance of POP on your system.
In the CCSM3.0 scripts, recommended values of NPROC_X and NPROC_Y are predefined for several processor configurations. The definition of NPROC_X and NPROC_Y is made in a two-step process: first, the scripts/$CASE/Buildexe/pop.buildexe.csh script sets the script variables NX and NY to appropriate values, based upon the number of tasks assigned to the ocean model. Second, these values of NX and NY are suppled as input to the models/bld/Macros.$ARCH macros file, where they are used to define the C preprocessor variables NPROC_X and NPROC_Y. The user can change these values by modifying the scripts/$CASE/Buildexe/pop.buildexe.csh to redefine NX and NY to the desired values.
If necessary, run the built-in benchmark problem (Appendix 7.3) on a variety of processor configurations (available in the standard LANL release only). See Appendix 7.4 for illustrative examples. (Standard LANL release only)
For some vertical and horizontal mixing parameterizations (e.g. Gent-McWilliams and KPP to be discussed later), large vertical mixing coefficients are used for both mixing and convection. In such cases, implicit vertical mixing must be enabled to avoid severe restrictions on the model time step. Because this option requires some code restructuring (breaking of a k-loop), the decision to use implicit vertical mixing must be made at compile time. The -Dimpvmix option should be uncommented in the archdir.gnu file to enable this option (standard LANL); by default, the -Dimpvmix option is enabled in the MACROS.$ARCH file (standard CCSM).
If POP is run in coupled mode, the default interface communicates with the NCAR CCSM Flux Coupler. This requires message-passing calls and linking with the MPI library (whether MPI is used internally or not). To enable this capability, the -Dcoupled option must be uncommented in the archdir.gnu file (standard LANL); the -Dcoupled option is enabled in the MACROS.$ARCH file (CCSM).