Skip to content

fix: count literal special tokens like <|endoftext|> as plain text - #129

Merged
anhnh2002 merged 1 commit into
mainfrom
fix/tiktoken-special-tokens
Oct 8, 2026
Merged

fix: count literal special tokens like <|endoftext|> as plain text#129
anhnh2002 merged 1 commit into
mainfrom
fix/tiktoken-special-tokens

Commits

  1. Commits on Oct 7, 2026