Tokens are wild and wonderful, but what about value?

I am endlessly curious, and sometimes I go down a bit of a rabbit or token hole. In this video, I take you down the whole with me so we can see, roughly, how deep the hole goes ;p Also, how do you determine whether your token spend equates to product value? What are your strategies for driving down token cost while increasing value? Let me know in the comments. OpenAI Tokenizer — https://platform.openai.com/tokenizer — exact for OpenAI encodings only. Tiktokenizer — https://tiktokenizer.com/ Anthropic token counting endpoint — https://platform.claude.com/docs/en/b... OpenAI Cookbook: How to count tokens with tiktoken — https://cookbook.openai.com/examples/... Karpathy, "Let's build the GPT Tokenizer" —    • Let's build the GPT Tokenizer  . Simon Willison's writeup on the above — https://simonwillison.net/2024/Feb/20... The minbpe repo is at https://github.com/karpathy/minbpe. Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies: https://arxiv.org/abs/2407.13623