← freeCAM docs
ne16 · PI-atm · online coupled · one month (1488 steps) · 4 A100 nodes

The fewer the ranks, the closer the model comes to the Fortran

Total time for one month

Seconds; H includes its one-time start-up

Original Fortran (CESM, A)Batched model (H)

Total time for one month: how much slower the model is

H relative to A; negative means the model is faster

One-time start-up cost

Seconds H takes beyond C, all in step 1

Data: one month

Half a year: 512 ranks

Half a year at 512 ranks (8688 steps, job 7689590): the model is still 16% slower, about 40 ms more per step, so a longer run never catches up.

LengthOriginal A (s)Model H (s)H/AH start-up (s)Original step (ms)Model step after start-up (ms)Forward (ms)
One month (7648576)496.2573.21.1645.4333.5355.247.8
Half year (7689590)2673.43095.71.1647.1307.7351.252.3

To beat the original, the one forward per step has to drop from about 52 ms to about 12 ms (the per-step time of the Fortran it replaces).

What the extra ~40 ms per step is made of

The 512-rank half-year run, per rank and per step. The first three rows add up to the 52.3 ms of one forward.

ComponentPer step (ms)Source
Model forward: cost with one rank alone+8.2CPU launching 793 small GPU kernels 4.0, TorchScript preparation 3.7, copies and waits 0.5 (standalone test)
Model forward: extra from 32 ranks sharing one GPU+33.641.8 − 8.2 (standalone test)
Model forward: extra in production over the standalone test+10.552.3 − 41.8, not broken down
Fetching results and checking inputs, per chunk+2.4
Gathering inputs, the rest of the shallow-convection Fortran+4.3
Original shallow convection, no longer computed−20.4Per-step time of the whole shallow convection in C, one-month run
Other (differences outside shallow convection, measurement noise)+4.9Remainder
Total: model over original, per step+43.5351.2 − 307.7 (half-year run)

Time of one forward

ms, 27 columns

0 ms 10 ms 20 ms 30 ms 40 ms 50 ms 8.2 1 8.3 2 8.6 4 8.6 8 12.6 16 41.8 32 Ranks calling the model at once on one A100 (32 in production)

The cost is the number of kernels: 793 small kernels per forward, about 25,000 for 32 ranks; the GPU computes for only 3 ms.

What the one-time ~47 s start-up is made of

Step 1's start-up redone on its own (job 7692487): the largest part is creating the CUDA context, 38 s when 32 ranks share each GPU.

Start-up time by part

Median per rank, seconds

Create CUDA context, connect to MPSLoad model, weights onto GPUWarm-up forward (16 all-zero columns)First 27-column forward
0 s 10 s 20 s 30 s 40 s 50 s 60 s 1 rank alone Create CUDA context, connect to MPS: 0.27 s Load model, weights onto GPU: 0.11 s Warm-up forward (16 all-zero columns): 0.62 s First 27-column forward: 0.16 s 1.2 s 8 per GPU (the 128-rank run) Create CUDA context, connect to MPS: 6.58 s Load model, weights onto GPU: 0.12 s Warm-up forward (16 all-zero columns): 0.69 s First 27-column forward: 0.17 s 7.6 s 32 per GPU (512-rank production) Create CUDA context, connect to MPS: 38.07 s Load model, weights onto GPU: 2.55 s Warm-up forward (16 all-zero columns): 12.75 s First 27-column forward: 0.59 s 54.0 s
Per rank (s, median)CUDA contextLoad modelWarm-upFirst 27-col forwardTotalLater forwards (ms)
1 rank alone0.270.110.620.161.28.2
8 per GPU, 32 total6.580.120.690.177.68.6
32 per GPU, 128 total (production)38.072.5512.750.5954.036.6