Seconds; H includes its one-time start-up
H relative to A; negative means the model is faster
Seconds H takes beyond C, all in step 1
Half a year at 512 ranks (8688 steps, job 7689590): the model is still 16% slower, about 40 ms more per step, so a longer run never catches up.
| Length | Original A (s) | Model H (s) | H/A | H start-up (s) | Original step (ms) | Model step after start-up (ms) | Forward (ms) |
|---|---|---|---|---|---|---|---|
| One month (7648576) | 496.2 | 573.2 | 1.16 | 45.4 | 333.5 | 355.2 | 47.8 |
| Half year (7689590) | 2673.4 | 3095.7 | 1.16 | 47.1 | 307.7 | 351.2 | 52.3 |
To beat the original, the one forward per step has to drop from about 52 ms to about 12 ms (the per-step time of the Fortran it replaces).
The 512-rank half-year run, per rank and per step. The first three rows add up to the 52.3 ms of one forward.
| Component | Per step (ms) | Source |
|---|---|---|
| Model forward: cost with one rank alone | +8.2 | CPU launching 793 small GPU kernels 4.0, TorchScript preparation 3.7, copies and waits 0.5 (standalone test) |
| Model forward: extra from 32 ranks sharing one GPU | +33.6 | 41.8 − 8.2 (standalone test) |
| Model forward: extra in production over the standalone test | +10.5 | 52.3 − 41.8, not broken down |
| Fetching results and checking inputs, per chunk | +2.4 | |
| Gathering inputs, the rest of the shallow-convection Fortran | +4.3 | |
| Original shallow convection, no longer computed | −20.4 | Per-step time of the whole shallow convection in C, one-month run |
| Other (differences outside shallow convection, measurement noise) | +4.9 | Remainder |
| Total: model over original, per step | +43.5 | 351.2 − 307.7 (half-year run) |
ms, 27 columns
The cost is the number of kernels: 793 small kernels per forward, about 25,000 for 32 ranks; the GPU computes for only 3 ms.
Step 1's start-up redone on its own (job 7692487): the largest part is creating the CUDA context, 38 s when 32 ranks share each GPU.
Median per rank, seconds
| Per rank (s, median) | CUDA context | Load model | Warm-up | First 27-col forward | Total | Later forwards (ms) |
|---|---|---|---|---|---|---|
| 1 rank alone | 0.27 | 0.11 | 0.62 | 0.16 | 1.2 | 8.2 |
| 8 per GPU, 32 total | 6.58 | 0.12 | 0.69 | 0.17 | 7.6 | 8.6 |
| 32 per GPU, 128 total (production) | 38.07 | 2.55 | 12.75 | 0.59 | 54.0 | 36.6 |