max-num-batched-tokens
Flag for max tokens processed per step; balances prefill vs decode and tail latency.
Flag for max tokens processed per step; balances prefill vs decode and tail latency.
Flag for max tokens processed per step; balances prefill vs decode and tail latency.
Flag for max tokens processed per step; balances prefill vs decode and tail latency.