Optimal Allocation
For a given compute budget C, the best way to split resources between model size (N) and data size (D) to minimize loss.
For a given compute budget C, the best way to split resources between model size (N) and data size (D) to minimize loss. The paper found: N_opt ∝ C^0.73, D_opt ∝ C^0.27.