UCAR |   NCAR |   CESM® : : Community Earth System Model

LIWG Developers' Guidelines

Current status

We are currently working on a beta version of CISM2.0. In this beta version, the development process has been relatively uncontrolled. Once CISM2.0 is released, tighter controls on code changes will be put in place; the rest of this document describes the process that we are moving towards.

 

Background

The ice sheet model code in CESM is split across two repositories: the glimmer-cism repository and the CESM repository. The glimmer-cism repository contains all code and build scripts needed to build a standalone version of CISM (i.e., outside the framework of CESM or any other climate model). Standalone CISM includes its own test suite. Most development is first carried out here, with initial testing done on the standalone model.

The CESM repository pulls in all code from the glimmer-cism repository (as an svn external). CESM has its own test suite, which tests the integration of the glimmer-cism code with CESM in various configurations. An implication of this configuration is that changes from the glimmer-cism repository can be pulled in to CESM at any time – changes to the glimmer-cism trunk do not need to be immediately incorporated into the CESM trunk. This setup could potentially allow the CESM trunk to have tighter change controls than the glimmer-cism trunk.

 

Scientific approval process, and basic development process

Any significant change requires scientific approval from the LIWG. This includes both (a) significant answer changes to default model configurations or other important model configurations, and (b) significant new features.

Code introducing new features to the default configuration is allowed to be merged with the trunk under the following circumstances and pending approval by the LIWG:

  1. New code adds significant, new and novel scientific capabilities

  2. New code adds significant improvements in computational and/or numerical performance

  3. New code improves model diagnostic output (e.g., with respect to observations)

The above items will be assessed using standard test cases that verify and validate model output and performance, relative to the current version of the code on the trunk.

We envision two types of development; the development process will differ for these two types:

  1. Development that the LIWG decides ahead of time is a high priority and should be done. This development will likely make it to the trunk eventually, unless it is found to have major flaws. This type of development should involve scientific and software liaisons throughout the development process, and especially in the early design phases where this involvement is most helpful. Assuming the liaisons have been consulted throughout development, only minor revisions will generally be needed when it comes time to bring the code to the trunk.

  2. More exploratory development, which a collaborator takes the initiative to undertake. For changes that the developer hopes to bring to the trunk eventually, the developer is still strongly encouraged to consult with scientific and software liaisons in the early design phases. (Even a small amount of design discussion can save significant time later.) However, because the liaisons’ time is limited, this type of development will probably have only limited input from scientific and software liaisons throughout the rest of development. As the development nears completion, the developer will run simulations showing that the development is correct and improves the model in one of the ways laid out above. At that point, the LIWG will decide whether this development should be brought to the trunk. Any code developed in this manner should be viewed as a prototype, which may need to be rewritten substantially before it can be brought to the trunk, in order to ensure that it follows existing coding standards and keeps the code base maintainable. In general, the burden for rewriting the code will fall on the original developer. However, scientific and software liaisons can lend their support here, with the level of support determined by the priority of the development to the LIWG.

Note that some kinds of changes do not require scientific approval. This includes:

  1. Minor bug fixes that do not change answers significantly, or change answers only in seldom-used configurations

  2. Code refactoring that changes answers by no more than roundoff

  3. Cosmetic code changes (which should not change answers at all)

However, depending on the nature of the change, the developer may be required to demonstrate that answers do not change significantly, e.g., by showing differences between a new and old simulation.

 

Software engineering approval process

All changes are subject to approval by the software engineering gatekeeper, including minor changes that are not subject to scientific approval. However, for minor changes (especially cosmetic changes that do not change answers), the gatekeeper may decide that a review is unnecessary. The gatekeeper’s role in this process is to ensure that all changes follow coding best practices and conventions (these will be articulated in a separate document), as well as to ensure that the code evolves in a maintainable way. In addition, the gatekeeper should evaluate whether new code has any undesirable interactions with other parts of the system, which may not have been considered by the original developer.

For small problems identified by the gatekeeper, the gatekeeper will work with the developer to fix these problems. However, for larger problems, including problems with the overall design of the new code, the gatekeeper may return the code to the developer for revision, with guidance as to what changes would be needed for this code to be acceptable for the trunk.

Developers are strongly encouraged to consult with the software engineering gatekeeper and other experienced members of the LIWG before undertaking any major developments that they would eventually like to bring to the trunk. Early design discussions can save significant time later, by preventing the need for a substantial rewrite of the code. Code that is developed without initial design discussions and periodic ongoing discussions with other developers should be viewed as a prototype. This prototype code may require substantial revisions before being accepted.

 

Trunk policy

Code must be well-tested before it can come to the main development trunk. This testing should include at least the standard test suite. (There is one test suite for the standalone glimmer-cism repository, and another for the CESM repository.) In addition, developers should perform any additional tests that they feel are required to ensure that their new features are working as intended. Developers, in consultation with the software engineering gatekeeper, should consider whether these additional tests should be incorporated into the standard test suite, to ensure that their new features continue to work in the future.

Code in the glimmer-cism repository does not necessarily have to undergo CESM testing before being brought to the trunk – the standalone test suite is sufficient. The CESM test suite will be run before the CESM repository is updated to include the latest glimmer-cism code.

Trunk commits are envisioned as happening every 1 – 2 weeks. In general, each commit to the trunk (either the glimmer-cism trunk or the CESM trunk) will be associated with a unique trunk tag.

 

Development branches

Development branches are generally used for two purposes:


  1. Development of new features: any change of more than a few lines can warrant the use of a feature branch.

    1. Developers are strongly encouraged to periodically update their branch to the latest trunk version. (A separate document will provide guidance about this.)

  2. As a holding area for minor, trunk-bound changes: the purpose of these branches is to avoid needing to run the full test suite on every minor change. These just apply to the glimmer-cism repository, not the CESM repository.

    1. For example, if the last trunk tag was cism2_0_00, then there could be a branch named to_be_2_0_01T (where the final T stands for “trunk”).

    2. A limited set of developers (including the software engineering gatekeeper, as well as LIWG co-chairs and liaisons) can commit minor changes to these branches directly. Others can commit minor changes with permission.

    3. In general, these branches should only include bit-for-bit changes. Answer-changing modifications generally warrant their own trunk tag, so that answer-changing modifications don’t mask problems with changes that are intended to be bit-for-bit.

    4. Larger commits to these branches may still require running a small subset of the test suite to catch any problems early

    5. We aim to bring these branches to the trunk every 1 – 2 weeks.


The line between what size development warrants its own branch (which will become its own trunk tag) vs being small enough to go onto a shared, “holding-area” branch is somewhat blurry. This will need to be determined from experience. However, note that answer-changing modifications (even roundoff-level) almost always warrant their own trunk tag.

 

Testing Responsibilities

For feature branches – which generally involve a single developer or a small, coordinated group of developers – the developers are responsible for running the standalone CISM test suite, and getting all tests to pass.

For “holding area” branches, the software engineering gatekeeper is responsible for testing. However, (s)he may require that people checking in changes to this branch run a small subset of the standalone test suite before making any commits.

The software engineering gatekeeper is responsible for incorporating the glimmer-cism code base into the CESM repository, and for running the CESM tests when this occurs. However, since the CESM tests often catch problems in the glimmer-cism code that were not caught by the standalone test suite, it is the responsibility of the original developers to fix any problems that are revealed in this testing.

 


Summary of process for bringing a new development to the trunk

The following summarizes the process for bringing a major new development to the trunk. Note that many of these steps should actually be performed periodically, earlier in the development process. For example, developers should periodically update their branch to the trunk, and should periodically run the test suite rather than saving testing for the end of the process when it is hard to determine what changes have caused a test to fail. In addition, it is helpful to involve the software engineering gatekeeper and other developers earlier in the process to avoid the need for major revisions.

  1. Get scientific approval to bring this development to the trunk

  2. Update branch to latest version of trunk

  3. Run standalone test suite, and any other tests that the developer feels are necessary to verify new functionality

  4. Software engineering gatekeeper reviews code. If major revisions are needed, the developer will have to make these revisions and then repeat steps 2 & 3, followed by a re-review.

  5. Gatekeeper runs CESM test suite

  6. Once standalone test suite & CESM test suite pass, gatekeeper checks in to trunk, makes trunk tag.