General AI
Explainer: How AI Models Use Tokens
Lev Selectorhigh confidence
Why it matters
- →Provides foundational understanding of AI text processing.
- →Clarifies the role of tokens in language models.
- →Helps demystify AI service cost structures.
A recent visual explainer video delves into the process of tokenization in AI models. It describes how text input is broken down into tokens, each assigned a numerical Token ID, which are then mapped to numerical vectors for neural network processing. The video also explains the difference between tokens and words, the significance of token counts in relation to a model's context window, and why some AI services measure usage in tokens. This foundational understanding aids in grasping more complex concepts like embeddings and AI-generated responses.
Read originalThe story around this
TopicAI Model Architecture And MechanicsCooling
Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.
How we got here
- 1New Paper Explores AI Negation Neglect
AI Explained · May 20, 2026 · Background
- 2AI Agents Learn to Ask Better Questions with Games
MIT News AI · June 3, 2026 · Background
- 3Hybrid Models Show Strength in Predicting Meaningful Tokens
Hugging Face Blog · June 25, 2026 · Related
- 4Thinking Machines Launches Inkling Multimodal LLM
Hugging Face Blog · July 15, 2026 · Related
- 5Together AI launches Inkling model on day 0
Together AI Blog · July 15, 2026 · Background
- 6Explained: How AI Transforms Text Prompts into Images
Lev Selector · August 3, 2026 · Related
- 7Understanding AI Tokens and Cost Management
The AI Daily Brief · August 4, 2026 · Related
- 8Explainer Video on AI Attention Mechanism Released
Lev Selector · August 12, 2026 · Related
What happened next
- 9Grok Bot Simplifies AI Agent Adoption
The AI Daily Brief · August 13, 2026 · Background
- 10Qwen3.8-27B Model Explored and Optimized
Sam Witteveen · August 18, 2026 · Background
- 11AI Models Struggle with Classic Intelligence Tests
MIT Technology Review AI · August 26, 2026 · Background
- 12Skild AI Unveils S1 Model for Task Learning
Matt Wolfe · September 3, 2026 · Background
- 13AI Revolutionizes Multimedia Content Processing
AI News · September 14, 2026 · Background
Follow this story
Open the full story →Explainer: How AI Models Use Tokens
2 developments
- Aug 12 · Lev Selector
Explainer: How AI Models Use Tokens (This article) - Aug 12 · Lev Selector
Explainer: How AI Embeddings Work in Language Models↳ AI embeddings convert text into numerical representations for neural network processing.
More in General AI
© The Verge AIGeneral AIother
Grokipedia resumes article updates after months-long pause
SpaceXAI’s Grokipedia is finally moving again after a six-month dormancy that left many pages stale and references unchecked. The platform is now actively fact-checking high-profile pages like Barack Obama’s and Elon Musk’s, while also accepting user-suggested edits for niche topics like video game releases. This shift from silence to activity suggests the team is prioritizing content freshness over the initial launch hype, though the inconsistency across pages indicates a gradual rollout rather than a full system overhaul.
The Verge AI
© The Verge AIGeneral AIother
AI Researchers Warn of Existential Risk in New Video Series
Palisade Research has launched a video series featuring insiders from OpenAI, Google DeepMind, and Anthropic warning that superintelligence poses an existential threat comparable to a coin flip for human extinction. The project brings internal moral conflicts into the open, with researchers like Daniel Kokotajlo describing superintelligent AI as 'god-like' and uncontrollable. Neel Nanda of Google DeepMind estimates a 10 percent chance of extinction, calling that figure 'ridiculously high.' These voices are not distant academics but employees who feel compelled to stay in their roles to mitigate risks they helped create. The series strips away the corporate veneer, revealing a community deeply divided on whether current safety measures are sufficient. This public airing of internal doubts shifts the conversation from abstract theory to urgent industry self-reflection.
The Verge AI
© TechCrunch AIGeneral AIother
White House launches AI chatbot for government services
The White House is deploying America.gov, an AI-powered front door for federal information, leveraging Google’s Gemini and xAI’s Grok to cut through bureaucratic complexity. This marks a significant shift in how citizens interact with the state, moving from static web portals to conversational interfaces. However, the reliance on generative models introduces tangible risks; unlike military hallucinations that nearly triggered conflict, errors here could directly impact visa renewals, tax filings, and benefit eligibility. The experiment tests whether LLMs can be trusted with high-stakes civic guidance without overwhelming human oversight.
TechCrunch AI