Before a process first uses the GPU, it creates its own CUDA context, once per process. This page goes from there, through the one-time start-up and how libtorch runs one forward, to CUDA Graph, compiled fusion and TF32.
What it does: opens this process's own workspace on the GPU (memory address space, GPU code, command queues) and connects to the MPS server.
Why it is needed: without it the process can neither put data on the GPU nor compute there; connected to MPS, it can compute at the same time as the other processes on the same GPU.
What it does: reads the TorchScript file (network structure and trained weights) and copies the weights to GPU memory.
Why it is needed: the weights stay in GPU memory and every later step uses them directly, without reading them again.
What it does: runs one forward on an input of the right shape filled with zeros and discards the result, so the model's results are unaffected.
Why it is needed: a model's first forward optimises the graph, loads the GPU operators it uses for the first time, and allocates the memory pool. The warm-up moves this before step 1, so step 1 is not suddenly more than ten seconds slower.
What it does: step 1 computes with real data for the first time.
Why it is slower than later ones: it is not a dedicated preparation step. The warm-up used 16 columns and a real step has 27; the shape changed, so it prepares once more for the new shape.
Measured, job 7692487: 1 GPU node (4 A100s), all processes starting at the same moment, median per process.
The graph shows 12 representative operators; the real one has 793. Times are measured: the start-up from job 7692487 (32 processes per GPU), each call from job 7692425 (1 GPU node), and 52.3 ms is the forward per step in the half-year production run 7689590.
Measured: the times with Graph come from job 7752711, the 8.2 and 41.8 ms without Graph from job 7692425 (both on one A100, 27 columns a call; 7752711 re-measured 32 processes without Graph at 40.2 ms). The 12 operators in the drawing stand for the real 793.
Measured: jobs 7752711 and 7755073 (one A100, 32 processes, 27 columns a call); the month's numbers come from jobs 7689590, 7749460 and 7755194.