SentencePiece
A subword tokenizer that converts text into tokens using a learned vocabulary.
A subword tokenizer that converts text into tokens using a learned vocabulary. Gemini uses SentencePiece with 256K vocabulary (shared across all models).
A subword tokenizer that converts text into tokens using a learned vocabulary.
A subword tokenizer that converts text into tokens using a learned vocabulary. Gemini uses SentencePiece with 256K vocabulary (shared across all models).