DEV Community

#tokenization

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Did Your AI Memorize Your Rules — or Understand Them? A Tokenizer Showed Me the Difference
icophy profile

Did Your AI Memorize Your Rules — or Understand Them? A Tokenizer Showed Me the Difference

#ai #llm #tokenization #promptengineering
1
Comments 1
4 min read
Your Gemini 3.8 Flash Token Counter Is Wrong
robust_true_try profile

Your Gemini 3.8 Flash Token Counter Is Wrong

#python #ai #gemini #tokenization
Comments
4 min read
Why a Small Transformer Can't Copy a Word It Hasn't Seen
megapixel99 profile

Why a Small Transformer Can't Copy a Word It Hasn't Seen

#measurement #codegeneration #llm #tokenization
Comments
10 min read
Tokens: The Currency of AI
internals_decoded profile

Tokens: The Currency of AI

#ai #tokens #tokenization #llmpricing
Comments
10 min read
5 Data & Analytics Tools Every Developer Building on Hyperliquid Should Know (2026)
chen_joseph_68dd2aadd8c54 profile

5 Data & Analytics Tools Every Developer Building on Hyperliquid Should Know (2026)

#hyperliquid #api #cryptocurrency #tokenization
1
Comments 1
2 min read
Ethereum ou Stellar: Quem Vence a Tokenização de Ativos no Brasil? [PT-BR]
andr_diasmoreiraprol_b profile

Ethereum ou Stellar: Quem Vence a Tokenização de Ativos no Brasil? [PT-BR]

#stellar #ethereum #blockchain #tokenization
1
Comments
4 min read
Tokenization under the hood: BPE, WordPiece, SentencePiece, and Unigram compared
tech_nuggets profile

Tokenization under the hood: BPE, WordPiece, SentencePiece, and Unigram compared

#tokenization #llm #ai #nlp
Comments
9 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.