Compute-Optimal Frontier
The boundary of efficient training allocations: the curve of (N, D) pairs that minimize loss for a given compute budget C.
The boundary of efficient training allocations: the curve of (N, D) pairs that minimize loss for a given compute budget C. For a fixed budget, you can't do better than points on this frontier; points above or to the side are suboptimal.