5 min read
Reading token offsets
What a character span in a tokenizer output actually points at, why [CLS] and [SEP] carry [0,0], and how offsets survive accent stripping.
ReadTagged output format
1 guide on output format.
1 guide on output format.
What a character span in a tokenizer output actually points at, why [CLS] and [SEP] carry [0,0], and how offsets survive accent stripping.
Read