TP Waste Benchmark v0.1
Structural waste measurement across GPU configurations for tensor parallel inference.
Efficiency = how much of theoretical speedup is actually delivered.
Structural waste = TP is slower than a single GPU.
productive
marginal
structural waste
27B model
| Configuration | DFlash | Speedup | Waste | Efficiency | Verdict |
|---|
70B model
key findings
| Configuration | DFlash | Speedup | Waste | Efficiency | Verdict |
|---|
DFlash + PCIe = structural waste on 27B models.
RTX5090, RTX PRO 6000, A100 PCIe, H100 PCIe all produce structural waste with DFlash enabled on 27B. AllReduce communication cost per draft token across 64 layers exceeds the compute savings from splitting the model.
NVLink changes everything.
2x RTX PRO 6000 NVLink goes from structural waste (0.74x) to 93% efficiency (1.93x) by switching interconnect. Same GPUs, same model, same DFlash setting. The interconnect is the deciding variable.
Larger models are more forgiving.
On 70B, most configurations that produce structural waste on 27B become marginal or productive. More compute per layer means AllReduce overhead is proportionally smaller.
Mixed GPU pairs can outperform identical pairs.
2x RTX4090 + RTX5090 on 70B produces 2.33x — 133% efficiency — outperforming both 2x RTX4090 (1.90x) and 2x RTX5090 (1.83x). Proportional weight splitting means the faster GPU carries more compute.
8x H100 NVLink remains productive at scale.
70B no DFlash delivers 7.13x speedup at 87.6% efficiency. NVLink at scale maintains near-linear scaling across all configurations tested.