KV-cache quantization
Storing the KV cache in 8-bit (or lower) to roughly halve its memory, enabling more concurrency / longer context.
Storing the KV cache in 8-bit (or lower) to roughly halve its memory, enabling more concurrency / longer context.
Storing the KV cache in 8-bit (or lower) to roughly halve its memory, enabling more concurrency / longer context.
Storing the KV cache in 8-bit (or lower) to roughly halve its memory, enabling more concurrency / longer context.