Power and energy during training

Qwen3 8B model · FSDP 4 + 4

Two servers train one model together: 4 GPUs on ml7 and 4 on ml1, linked at 100 Gb. Power is how fast energy is used right now; energy is how much has been used so far.

1.61 Jenergy per token, both whole servers

84 sper training step

2.2 kWboth servers together, on average

The whole run

The box marks the 10 seconds replayed below.

0.00 s

Server map

Show the meters

Power mode: each part glows brighter the more power it draws right now, on one scale for every part.

GPU outline: computing computing and communicating waiting for weights waiting to pass gradients idle or not used

data inside a server data to the other server

GPU states

What each GPU is doing, from the Nsight trace. The middle line is now; the bars show one second either side.

Power over time

Dots are the meters’ readings. The height of the line is power; the shaded area up to now is the energy used so far.

Whole-server power

Each bar is one server’s power right now, split into its parts. The white tick is the wall-plug reading.

Run details

Model	Qwen3 8B, pretrained weights
Parallelism	FSDP (fully sharded data parallel) over 8 GPUs: ml7 GPUs 0–3 (Quadro RTX 6000) + ml1 GPUs 0–3 (RTX A5000)
Link between servers	100 Gb RDMA (RoCE)
Precision	fp16
Dataset	C4 (c4_test)
Sequence length	2,048 tokens
Batch	56 sequences per step (114,688 tokens)
Steps	5 (steps 2–5 measured) · activation checkpointing: full · seed 42
Date	31 Oct 2026
Shown here	10 seconds from the middle of step 3 (of 84 s), replayed
Meters	GPUs: their own meters, every 20 ms · CPU and memory: CPU counters, every second · whole server: wall plug, every second · GPU states: Nsight Systems trace