7 min read
Counting against a context window
Measured tokens-per-word ratios for plain prose, URLs, code and CJK, and why budgeting a prompt from word count alone is wrong.
ReadGuides
Each guide explains one thing about tokenizer output, using counts measured against the exact vocabularies this site pins. Every number can be checked in the tool.
5 min read
Reading token offsets
What a character span in a tokenizer output actually points at, why [CLS] and [SEP] carry [0,0], and how offsets survive accent stripping.
Read 6 min read
How WordPiece splits text
The greedy longest-match rule behind BERT subword tokens, why continuations carry
Read 6 min read
Diagnosing [UNK] tokens
What an [UNK] token actually means, why one unmatchable character costs a whole word, and how to choose a tokenizer that fits your input.
ReadThe ids of [CLS], [SEP], [MASK], [PAD] and [UNK], and what each one is for
Five control tokens, five fixed ids, five different jobs in a BERT forward pass. What each one does in the sequence, when it is added, and how to tell them apart from the token you meant to type.
ReadWhat tokenizer.json and tokenizer_config.json each decide
One file holds the vocabulary and the algorithm, the other holds the switches that decide what happens before lookup. Confusing the two produces a tokenizer that loads cleanly and counts wrong.
ReadWhy the line number in vocab.txt is the token id
A WordPiece vocabulary is an ordered list, so a token's id is its position. That single fact explains why reordering lines silently corrupts a checkpoint and why adding a word is not an editing task.
ReadWhy the special tokens are not in the WordPiece vocabulary file
[CLS], [SEP], [MASK], [PAD] and [UNK] live outside the main vocabulary, which is why pasting them into a BERT tokenizer gives you ordinary subword fragments instead of the control tokens you typed.
ReadWhich of the two vocabularies to use
Measured side by side on English prose, mixed case, accents, URLs and CJK, so the choice between bert-base-uncased and bert-base-multilingual-cased is made on counts rather than on which model name sounds more capable.
Read