TokenAtlas 🌐

The universal benchmark quantifying the token inflation tax across 52 global languages, code, and agent tool schemas across frontier LLM tokenizers.

⭐ Star on GitHub 🤗 Hugging Face 13 Frontier Tokenizers 52 Languages (FLORES-200) Multi-Domain Benchmark Apache 2.0 Open Source

🔬 Interactive Visual Tokenizer Playground & Live Token Tax Inspector

Type or paste any text or code to inspect how different frontier tokenizers partition the sequence into color-coded token chips and byte fragments.

0 chars | 0 bytes
Presets:
0
Tokens
-- bytes/tok Native Merges
0
Tokens
-- bytes/tok --