Blocks

The size of the blocks for which the domain is decomposed is set by two parameters in the domain_size module (contained in the file domain_size.F90 in the run directory). The parameter block_size_x determines the number of physical grid points in the first horizontal dimension; the parameter block_size_y determines the number of physical grid points in the (wait for it...) second horizontal dimension. The two parameters max_blocks_clinic and max_blocks_tropic determine the maximum number of blocks that can be distributed onto a processor or node. An initial guess for these two parameters can be made by dividing the total number of blocks by the number of processors you plan to run on

(nx_global/block_size_x)(ny_global/block_size_y)/nprocs (2.1)

Only an initial guess is required; when running the code, the actual numbers required will be output so that the user can set these parameters correctly. Also, note that these parameters can be set higher than required with no penalty other than memory use. For example, the parameters can be set for the lowest processor count you plan to use and then the same executable can be used to run at higher processor counts with no change other than the namelist inputs for number of processors.

As mentioned in previous sections, finding the optimal block size and distribution can require experimentation. However, a starting point for users who are familiar with previous versions of POP is to set the block size such that only one block is distributed on each processor. In this case, block_size_x = nx_global/NPROCX where NPROCX is the value found in the old POP .gnu files; values for the y direction are computed analogously. The max_blocks parameters are then set to one. After using this configuration, users can experiment with reducing the block size to improve performance.

CESM1 Notes

The CESM1 scripts use domain decomposition tools (located in the $CASE/Tools/Templates directory) to automatically set the CPP variables used to define the CESM1 POP2 block settings. These variables appear in the $CASE/Buildconf/pop2.buildnml.csh script on the gmake command line:

-DBLCKX=$BLCKX -DBLCKY=$BLCKY -DMXBLCKS=$MXBLCKS