← freeCAM docs

How a model forward runs on the GPU

Before a process first uses the GPU, it creates its own CUDA context, once per process. This page goes from there, through the one-time start-up and how libtorch runs one forward, to CUDA Graph, compiled fusion and TF32.

Why a process needs a context

Why crowding makes it slow

The one-time start-up cost

  1. Create the CUDA context, connect to MPS

    What it does: opens this process's own workspace on the GPU (memory address space, GPU code, command queues) and connects to the MPS server.

    Why it is needed: without it the process can neither put data on the GPU nor compute there; connected to MPS, it can compute at the same time as the other processes on the same GPU.

  2. Load the model

    What it does: reads the TorchScript file (network structure and trained weights) and copies the weights to GPU memory.

    Why it is needed: the weights stay in GPU memory and every later step uses them directly, without reading them again.

  3. Warm-up forward

    What it does: runs one forward on an input of the right shape filled with zeros and discards the result, so the model's results are unaffected.

    Why it is needed: a model's first forward optimises the graph, loads the GPU operators it uses for the first time, and allocates the memory pool. The warm-up moves this before step 1, so step 1 is not suddenly more than ten seconds slower.

  4. First real forward

    What it does: step 1 computes with real data for the first time.

    Why it is slower than later ones: it is not a dedicated preparation step. The warm-up used 16 columns and a real step has 27; the shape changed, so it prepares once more for the new shape.

Measured, job 7692487: 1 GPU node (4 A100s), all processes starting at the same moment, median per process.

How libtorch runs one forward

The graph shows 12 representative operators; the real one has 793. Times are measured: the start-up from job 7692487 (32 processes per GPU), each call from job 7692425 (1 GPU node), and 52.3 ms is the forward per step in the half-year production run 7689590.

CUDA Graph: record once, then replay

Measured: the times with Graph come from job 7752711, the 8.2 and 41.8 ms without Graph from job 7692425 (both on one A100, 27 columns a call; 7752711 re-measured 32 processes without Graph at 40.2 ms). The 12 operators in the drawing stand for the real 793.

Compiled fusion and TF32

Measured: jobs 7752711 and 7755073 (one A100, 32 processes, 27 columns a call); the month's numbers come from jobs 7689590, 7749460 and 7755194.