One hundred twenty-eight thousand tokens, and not one of them differed. That is what came back when I lined up a Korean tokenizer shipped by a Korean lab against Meta’s Llama 3, token by token.
I wanted to know whether Korean labs actually build their own rulers for Korean, so I pushed the same document through 12 tokenizers and counted. The conventional answer held up on one side of the table and broke on the other. Three of the newest Korean releases took the top three spots at 0.586 to 0.607 tokens per Hangul syllable. Two others sold under Korean branding turned out to carry base vocabularies identical to Llama 3 and Mistral, down to the sorted sha256 hash.
What the Korean tokenizer test actually measured
The corpus is the Universal Declaration of Human Rights, Korean and English, both official PDFs published by the UN Human Rights Office. One publisher, one document, two languages. That is about as close to a translation-equivalent parallel corpus as public text gets. After extraction the Korean version came to 4,813 normalized characters holding 3,344 Hangul syllables, the English version to 10,760, and all 30 articles verified on both sides.
The headline metric is tokens per Hangul syllable, with the denominator pinned at 3,344. Tokens per total character, spaces and punctuation included, sits in a separate column. cl100k_base reads 1.429 on the first metric and 0.993 on the second. Same text, same tokenizer, different denominator. Any percentage that does not name its denominator is noise, which is why both columns ship.
Every encoding ran locally, on tokenizer files pulled from public endpoints that need no authentication. Anything behind a login was dropped before measurement, not after.
Two extraction traps showed up along the way. Both would have quietly bent the table.
The UN English original numbers its first article in Roman, as Article I, then switches to Arabic for 2 through 30. A rule that looks only for digits drops article 1 and leaves you with 29 pairs instead of 30. That is exactly what happened on my first run, and the verification gate caught it. The second trap is Korean-specific. Hangul PDFs extract headings with letter spacing, and an aggressive rule for stitching those back together will also weld real word boundaries shut.
So I raised the threshold to fire only on runs of four or more single syllables, then checked afterward that the Hangul syllable count survived normalization unchanged. Raw and normalized figures both ship for the same reason.
The ranking, and where it stops being flattering
Twelve tokenizers, one corpus, side by side. Lower is better in the middle column, which means fewer tokens for the same Korean. The rightmost column converts that into how much Hangul fits inside a one-million-token window. Six of the twelve are Korean releases, four are global open models and two are OpenAI encodings.
| Tokenizer | Vocab size | Tokens per syllable | KO/EN token ratio | Hangul per 1M window |
|---|---|---|---|---|
| K-EXAONE-236B-A23B | 153,600 | 0.586 | 1.028 | 1,706,993 |
| A.X-4.0 | 102,400 | 0.589 | 0.813 | 1,697,461 |
| kanana-2-30b-a3b | 128,256 | 0.607 | 0.907 | 1,647,290 |
| Mistral-Small-24B | 131,072 | 0.783 | 1.136 | 1,277,310 |
| EXAONE-4.0-32B | 102,400 | 0.784 | 1.116 | 1,275,848 |
| o200k_base (GPT-4o) | 200,019 | 0.860 | 1.275 | 1,162,726 |
| kanana-1.5-8b | 128,259 | 0.873 | 1.292 | 1,145,990 |
| Llama-3.1-8B | 128,256 | 0.873 | 1.292 | 1,145,990 |
| Qwen3-8B | 151,669 | 0.914 | 1.341 | 1,093,525 |
| DeepSeek-V3 | 128,815 | 0.951 | 1.419 | 1,051,903 |
| cl100k_base (GPT-4) | 100,277 | 1.429 | 2.116 | 699,581 |
| SOLAR-10.7B-v1.0 | 32,000 | 1.520 | 2.009 | 658,008 |
Collected 2026-08-03 KST from each repository’s main branch. These figures come from the normalized extraction. On the raw text the top three reorder to kanana-2, K-EXAONE and A.X-4.0. The rightmost column assumes the same mix of spaces and punctuation as this corpus, where 3,344 of 4,813 normalized characters are Hangul. SOLAR-10.7B-v1.0, kanana-1.5-8b and EXAONE-4.0-32B are earlier Korean generations. These numbers move if a vendor swaps its tokenizer.
The top three are all recent Korean releases, bunched tightly at 0.586, 0.589 and 0.607. Convert that into working capacity and the spread turns concrete. A one-million-token window holds the equivalent of roughly 1.7 million Hangul characters of text like this document under K-EXAONE and roughly 700,000 under cl100k_base, a 2.4x difference on the same nominal window.
So far the conventional wisdom survives.
kanana-1.5 carries Llama 3’s vocabulary, hash and all
The next rows are where it broke. kanana-1.5 and Llama-3.1 both read 0.873, matching to the third decimal. Too many digits lined up for coincidence, so I opened the vocabularies and compared them directly.
Treating the 128,000 base tokens as sets, the difference in both directions came back empty. All 128,000 shared tokens hold the same numeric IDs. The sha256 of the sorted token set matches. Total vocabulary splits 128,259 against 128,256, and that gap is entirely special control tokens, four unique to kanana against one unique to Llama.
The kanana-1.5 model card names no base model. It says nothing about whether a tokenizer was trained or what size the vocabulary is. A llama tag sits on the Hugging Face page and that is the whole disclosure. What this measurement adds is resolution, a token-level verdict where the tag only offers a hint.
My read
The more interesting finding is the sequel. Kakao did replace the vocabulary one generation later, the model card says so explicitly, and the set comparison backs the claim.
- kanana-1.5 matches Llama-3.1 exactly on base vocabulary, with an empty difference, identical IDs and an identical hash.
- kanana-2 holds 60,848 tokens Llama-3.1 does not, and vice versa. Vocabulary size is the same 128,256, yet 60,848 of the 128,000 base tokens differ, or 47.5 percent.
- Measured on the same corpus, tokens per syllable fall from 0.873 to 0.607. Training a new vocabulary leaves a mark you can count.
SOLAR ships Mistral’s 32k. A.X-4.0 went the other way
The same test applied to SOLAR-10.7B-v1.0 returned the same verdict. Its 32,000 base tokens match Mistral-7B-v0.1 with an empty difference, identical IDs and an identical hash. Upstage disclosed the lineage, though. The model card states plainly that Mistral 7B weights were integrated into the upscaled layers. It says nothing about the tokenizer, but the provenance is not the news here. The consequence is.
SOLAR posts 1.520 tokens per Hangul syllable, the worst of the twelve. That is heavier than cl100k_base at 1.429, the encoding OpenAI shipped with GPT-4. With only 32,000 slots to work with, Korean shatters, and the shattering lands directly in the number.
Then A.X-4.0 ran the comparison backwards. Its model card credits Qwen2.5 as the base model. Yet the vocabularies overlap on just 38,628 tokens, and not one of those shared tokens sits at the same ID. A.X holds 63,772 tokens Qwen2.5 lacks; Qwen2.5 holds 113,015 that A.X lacks. SKT took someone else’s base model and then built its own ruler for Korean.
As a control I ran Qwen3-8B against Qwen2.5-7B. A full generation apart, and all 151,643 base tokens came back identical. The totals differ only by special tokens, 151,669 against 151,665.
Inheriting a base model’s tokenizer is ordinary practice in open weights, and neither model breaks a license by doing it. The mismatch between label and substance runs in opposite directions across these two cases. One wears domestic branding over an imported vocabulary. The other discloses an imported base while building the vocabulary in house. When I audited open-weight licenses earlier, the same class of gap showed up. A label does not vouch for what sits behind it.
Checking the vendors’ own numbers
The useful part of running everything through one instrument is that vendor claims become checkable. Each of the three Korean labs published efficiency figures, and none of them names the corpus or the baseline behind it, which is what normally makes the three incomparable.
| Vendor claim (primary doc) | Re-measured on one corpus |
|---|---|
| Kakao, kanana-2 Korean “improvement of over 30%” | 0.8726 to 0.6071, a 30.4% reduction (baseline unstated, pinned to the previous generation) |
| Kakao, kanana-2 vocabulary size 128,256 | 128,256, exact match |
| SKT, A.X-4.0 “approximately 33% fewer tokens than GPT-4o” | 0.8600 to 0.5891 against o200k_base, a 31.5% reduction |
| LG, K-EXAONE vocabulary size 153,600 | 153,600, exact match |
| LG, token efficiency “~30%” from a 6-language vocabulary redesign, not Korean-specific, baseline unstated | 0.7838 to 0.5858 against its own EXAONE-4.0, a 25.3% reduction |
All three claims point the right way, and both vocabulary sizes match to the digit. SKT’s reduction came in 1.5 percentage points below its stated figure. LG’s roughly 30 percent efficiency claim names no baseline at all, and pinning it to the company’s own previous generation produces 25.3 percent. All three re-measurements use the normalized extraction. On the raw text they read 29.4, 27.1 and 25.3 percent, which puts Kakao’s over-30% claim on one side of its own threshold and then the other.
Where Korean costs fewer tokens than English
Running the two language versions separately yields a ratio. On A.X-4.0 that ratio is 0.813, which means 1,970 tokens for the Korean text against 2,423 for the English. Korean coming out cheaper than English is a real regime, not a rounding artifact.
Crediting the tokenizer alone would be wrong, though. The same content occupies 4,813 characters in Korean and 10,760 in English. Hangul’s character density and tokenizer efficiency are stacked on top of each other in that single number.
The ratio also refuses to sit still. Cutting the document into its 30 articles and measuring each pair separately, each tokenizer’s largest per-article ratio runs 49.0 to 67.6 percent above its smallest. A.X-4.0 runs a median of 0.889 across a 0.694 to 1.075 band; cl100k_base runs a median of 2.220 across 1.780 to 2.652. Pick one short sentence, measure it, and declare that Korean costs some multiple of English, and the multiple you get depends on the sentence you picked. That is why the write-ups that rank for this query quote numbers that disagree with each other.
AI Compute Costs: The Great Divergence
What this measurement does not say
Three models never made the table because they refused to be measured. Google’s gemma-3-27b-it, Naver’s HyperCLOVAX-SEED-Text-Instruct-1.5B and Upstage’s solar-pro2 all sit behind access approval. Commercial closed models such as Claude and Gemini are out of scope entirely. Meta’s official Llama 3.1 repository is gated too, so that row comes from a mirror, and I cross-checked two independent mirrors against each other, confirming that the 128,000 base tokens match down to the hash before relying on either.
Vocabulary size proves nothing on its own. kanana-2 and Llama-3.1 both report 128,256, and 60,848 tokens still separate them. Matching sizes do not imply a shared vocabulary, and differing sizes do not rule out shared ancestry.
Tokenizer efficiency is also not model quality. This is a measurement of how text gets cut into pieces, and what happens after the cutting belongs to a different layer. The same caveat applied when I measured which open models actually get downloaded.
The row worth watching next is the one Kakao already wrote. A company that shipped someone else’s vocabulary in one generation and trained its own in the next moved from 0.873 to 0.607 on this corpus. If Upstage’s solar-pro2 and Naver’s HyperCLOVAX-SEED open their gates, the same instrument applies to them. Which side of the label they land on becomes the next line of this table.
References
- Hugging Face, skt/A.X-4.0 model card (accessed 2026-08-03) – Qwen2.5 base disclosed, approximately 33% fewer tokens than GPT-4o claimed, 131,072 context
- Hugging Face, kakaocorp/kanana-2-30b-a3b-instruct model card (accessed 2026-08-03) – newly trained tokenizer, vocabulary 128,256, over 30% Korean improvement claimed
- Hugging Face, kakaocorp/kanana-1.5-8b-instruct-2505 model card (accessed 2026-08-03) – no base model stated, apache-2.0
- Hugging Face, LGAI-EXAONE/K-EXAONE-236B-A23B model card (accessed 2026-08-03) – redesigned 150k vocabulary with SuperBPE, vocab size 153,600
- Hugging Face, upstage/SOLAR-10.7B-v1.0 model card (accessed 2026-08-03) – Mistral 7B weights integrated into upscaled layers, depth up-scaling
- GitHub, openai/tiktoken model.py (accessed 2026-08-03) – gpt-4o maps to o200k_base, gpt-4 and gpt-3.5-turbo map to cl100k_base
- OHCHR, Universal Declaration of Human Rights, Korean official PDF (accessed 2026-08-03) – measurement corpus, 4,813 normalized characters, 3,344 Hangul syllables
- OHCHR, Universal Declaration of Human Rights, English official PDF (accessed 2026-08-03) – measurement corpus, 10,760 normalized characters, article 1 written in Roman numeral
- Hugging Face, Qwen/Qwen2.5-7B-Instruct (accessed 2026-08-03) – vocabulary control for A.X-4.0 and Qwen3
- Hugging Face, mistralai/Mistral-7B-v0.1 (accessed 2026-08-03) – vocabulary control for SOLAR-10.7B, 32,000 base tokens
Stay Ahead of the Curve
Get the latest insights on AI, tech, and markets — delivered in both Korean and English.
Jugger | Backend Developer
A backend engineer working in enterprise solutions software since June 2021. The hands-on perspective from system design, API implementation, and production operations directly informs this site’s
AI, IT, and cloud infrastructure coverage.
For topics outside direct professional experience—blockchain and macroeconomics—analysis is built on primary-source citations such as SEC filings, whitepapers, institutional research, and government
statistics. Speculation is clearly separated from verified fact, and unverified claims are not published.
AI tools assist with drafting, but every article is reviewed, revised, and verified before publication. Errors are corrected with visible notes when discovered.
Contact: jjunggulee@gmail.com