freeCAM

Run the CAM Freely

CAM is the atmosphere component of the Community Earth System Model (CESM). However, its current workflow requires the model to be compiled into a Fortran executable and then submitted and queued on an HPC system. As a result, changing which process runs, or what computes it, requires editing the model, rebuilding it, and resubmitting it. Further, the entire case preparation process follows a Fortran style that is difficult to use. These limit the efficiency of scientific exploration.

To this end, we developed freeCAM, which drives the CAM atmosphere component from a Jupyter notebook. It supports deciding which process runs next and freely replacing a whole process or a single kernel with numerical code written in Python or a trained ML model, without stopping the running simulation.

Driving CAM from a Jupyter notebook

Figure 1 shows how a Jupyter notebook drives freeCAM on the NSF NCAR Derecho Supercomputer. Below the figure, we explain the notebook code line by line.

A Jupyter notebook starts one 512-rank MPI job on the NSF NCAR Derecho Supercomputer with initialize() and then sends it commands; on every rank, freeCAM in Python decides what runs and CAM in the original Fortran computes it
Figure 1. A Jupyter notebook starts one 512-rank MPI job on the NSF NCAR Derecho Supercomputer with initialize() and then sends it commands. On every rank, freeCAM (Python) decides what runs, and CAM (the original Fortran) computes it.
  1. import freecam as fc

    Imports freeCAM.

  2. d = fc.Driver(case="PI-atm")

    Creates a driver for the PI-atm case, i.e., the preindustrial atmosphere with the CESM land, sea ice, data ocean, and river components coupled online.

  3. d.initialize()

    Submits one job of 512 MPI ranks to Derecho and waits for every rank to connect. Every later cell reuses this job rather than submitting a new one.

  4. d.run(steps=48)

    Advances the model by 48 steps, i.e., one model day.

  5. d.cam.state.T.stats(rank="global")

    Computes the statistics of temperature over all 512 ranks and sends them back to the notebook.

  6. rad = d.processes["radiation"]

    Looks up the radiation process of this run.

  7. rad.process = my_network

    Puts my_network, e.g., a trained ML model, in place of the whole radiation process. The simulation is not rebuilt and the job is not resubmitted.

  8. d.run(steps=48)

    Runs one more model day, in which my_network computes radiation and every other process still runs in the original Fortran.

Besides Jupyter notebooks, freeCAM can also be driven ➊ from a Python script, ➋ as a PBS batch job, or ➌ from the Workflow Builder, a browser page we developed to edit, run, and watch the model. All of them reach the same Python control layer. Readers can refer to the Manual for how to use the Workflow Builder.

Bit-for-bit with the original

freeCAM with nothing replaced reproduces the original CESM bit for bit. We compare the five-year simulation of freeCAM with that of the original CESM. All 215 variables and 884 files written by the coupled run are identical, with no numerical tolerance. Figures 2 to 4 exemplify the state of a freeCAM run with different settings.

Temperature at 848 hPa at the start of a step, then the change made by each process in turn: vertical diffusion, gravity-wave drag, dynamics, deep convection, shallow convection, cloud macro/microphysics and radiation
1 / 8
Figure 2. Temperature at 848 hPa at the start of step 24, followed by the change made by each process in turn: vertical diffusion, gravity-wave drag, dynamics, deep convection, shallow convection, cloud macro/microphysics, and radiation.
Temperature at the lowest model level through 50 steps, one model day: the land warms by day and cools by night, the ocean barely changes
1 / 50
Figure 3. Temperature at the lowest model level over 50 steps, i.e., one model day. The land warms by day and cools by night, while the ocean barely changes.
The globe viewer: temperature at 848 hPa over the globe, the column picked at 23.5N 12.8E, and the latitude-pressure section along its meridian
Figure 4. The globe viewer, showing temperature at 848 hPa, the column picked at 23.5°N 12.8°E, and the latitude-pressure section along its meridian.

Close to the original with a trained network

We replaced the shallow-convection kernel, compute_uwshcu_inv, with a trained network inside the coupled model. The network runs on A100 GPUs, with land, sea ice, ocean, and rivers running live on CPU. As shown in Figure 5, its mean precipitation preserves the spatial pattern of the original (correlation 0.95) and its global mean (3.05 and 3.06 mm/day). That said, the network run produced 2.7% more low cloud and 5% more cloud liquid water. On A100 GPUs, the half year took 1.16 times as long as the original Fortran.

Jupyter notebookyour Python session
import freecam as fc
d = fc.Driver(case="PI-atm")
d.initialize()
shallow = d.processes["shallow_convection"]
net = fc.NativeModel("uwshcu.pt")
shallow.kernels["compute_uwshcu_inv"] = net
d.run(steps=8688)   # half a model year
Half-year mean total precipitation from the original Fortran and from the run with a network in compute_uwshcu_inv, and their zonal means: the patterns agree, global means 3.05 and 3.06 mm/day
Figure 5. Half-year mean total precipitation from the original Fortran and from the run with a network in compute_uwshcu_inv, together with their zonal means.

Readers can refer to the following pages for the accuracy, cost, and GPU details of this experiment.