Gigatoken is an MIT-licensed Rust BPE tokenizer that encodes text at gigabytes per second. On a 144-core AMD EPYC 9565, it runs GPT-2 tokenization at 24.53 GB/s. That is 989x HuggingFace tokenizers and 681x tiktoken, ag…
Open-weights releases pressure closed pricing and widen access, changing the build-vs-buy math for the whole ecosystem.
Summaries are aggregated for information only — follow the source link for the full story. Demo entries are illustrative.